CSC460 Theory of Computation (Spring 2005) 


Instructor 


e Nobo Komagata, komagata@tcnj.edu, Holman Hall 229, Office hours: See 
http://www.tcnj.edu/~komagata/schedule 


On-line Resources (required reading) 


e Course web page: http://www.tcnj.edu/~komagata/csc460/05s 
e The instructor’s information page for students (http://www.tcnj.edu/~komagata/students.html) 
e SOCS: http://socs.tcnj.edu/ (the following sections will be used for this course) 
o E-mail: Convenient to send messages to the class or a group of students. 
o Discussion: The SOCS discussion board will be used for certain types of announcements, question-answer, 
and discussion. In order to get informed of postings in a timely manner, students should turn on the 
Email Notification feature of the SOCS discussion board. 
o Grading: Used for posting module and course grades. Note that the “Grade” field will show “1” when the 
grade is available and the actual grade will appear in the “Note” field. 


Catalog Course Description 


This course focuses on the traditional, algorithmic theory of computation consisting of three subareas: (1) 
computability, (2) complexity theory, and (3) formal languages and automata. The topics include: Turing machines, 
decidability/undecidability, reducibility, Church-Turing thesis, context-free grammars/languages, push-down 
automata, finite automata, regular expressions/languages, and time/space complexity including NP-completeness. 
[Prerequisites: CSC310, CSC340] 


Learning Goals 


In this course, we examine a small number of general principles that lie behind a variety of computational 
phenomena so that we will be able to apply these principles to a broad range of real-world computation. One of the 
main goals of this course is to convince students that “theory” is useful for practically anyone. To do so, we need to 
observe examples of theory at work. Once we understand the benefits of using theory in general, we will be ready to 
explore Theory of Computation, which has always been behind the development of computer science. At the same 
time, we will also examine the limitations of Theory and identify what is really behind real-world computation. In 
this course, we will discuss most of the topics traditionally covered in an undergraduate-level Theory of 
Computation course. However, our approach will be to explore the possibility of applying Theory to practical 
problems which students are interested in. The course will not follow the textbook, nor will it do most of the 
textbook problems. 


Content Goals (mastery of the following core concepts, deep understandings, misunderstandings, and technical 
knowledge) 


1. Practical problems can often be transformed into research and computational problems. Every problem is 
associated with cost/significance, which is relative to the evaluator. Computational problems can be 
represented as a set, readily available as the input to computational processing. [problem] 

2. A theory is a potentially infinite, consistent body of knowledge which can be systematically derived from a 
small number of abstract principles. The gap between the principles and the entire information is the source of 
the theory’s predictive power. Also being abstract, a theory can be applied to a broad range of phenomena, 
which might appear distinct. [theory] 

3. Interactive computation subsumes algorithmic computation, but not vice versa. That is, there is a qualitative 
difference between these two modes of computation. [interactive computation] 

4. The algorithmic notion of computation can be represented in a variety of equivalent forms, which define a 
bounded class of sets. That is, there is a limit to algorithmic computation. [computability] 

5. The computable class of sets contain a hierarchy of proper subsets which can be characterized by distinct 

grammars and automata. [formal languages and automata] 

The practicality of an algorithm depends on its complexity relative to the input data size. [complexity] 

7. Power set, also encompassing the distinction between determinism and nondeterminism, can introduce 
discontinuity with respect to computability and complexity. [power set] 


a 


Performance Goals (expected outcomes and abilities to be observed as a result of successful learning) 


1. Identify real-world problems which are relevant to the student’s life and can be tackled by computational 
means. [awareness] 

2. Transform real-world problems into research problems, and then into computational problems, along with the 
analysis of the cost/significance of a problem. [transformation] 

3. Analyze computational problems with respect to interactivity, computability, language/automata hierarchy, and 
complexity hierarchy. Then, evaluate the analysis with respect to its usefulness, correctness, and accuracy. 
[analysis/evaluation] 

4. Respect, analyze, and give constructive criticisms to the ideas in the literature and those expressed by other 
people. [critical attitude] 

5. Express ideas orally and in writing, in a manner clearly understood by other students (with equivalent 
background). Explain your own ideas orally and in writing, clearly and logically. Revise the ideas, reflecting 
the feedback from other students and the instructor. [communication] 

6. Take initiative in both independent and group activities. Also extend the domain of theoretical inquiry beyond 
the scope prepared by the instructor. [initiative] 

7. Reflect upon the student’s own thinking process and assess the student’s own performance relative to the 
content and performance goals. [reflection] 


Course Modules 


This course is divided into the following four modules. 


Module A: Problems, Theories, and Computability (1) 
Module B: Computability (2) 

Module C: Formal Languages and Automata 

Module D: Complexity 


However, the topics of this course cannot be clearly separated into disjoint components. Thus, these modules should 
not be considered rigidly. Instead, these modules should be considered as evaluation units so that students can 
check their achievements in a timely manner. 


Assessment 


This course is expected to facilitate learning experience with which students can analyze practical computational 
problems in a intellectual manner by applying principles in Theory of Computation. Therefore, students’ assessment 
must reflect their ability to do that in a realistic context. Take-home exercises are designed to provide activities 
reflecting the learning goals (except for Performance Goal 8) in a realistic context. Thus, students’ assessment will 
primarily based on how well they accomplish these exercises. As for Performance Goal 8, evaluation workshops 
will provide opportunities for students to reflect upon their thought processes and self-evaluate their achievements 
with respect to the learning goals. 


The main assessment tools are students’ self-evaluation at the end of each module. Students will evaluate 
themselves with respect to the learning goals following the instructions on the self-evaluation form (preliminary 
eval form for Module A). Since students will be given the form at the beginning of each module, they will be 
aware of what they are expected to achieve for that module. The evaluation form will list the learning goals relevant 
to the module as well as evaluation criteria for the relevant goals. Analysis and reflections on take-home exercises 
will be most important as the exercises address the learning goals directly or indirectly. Students are also expected 
to respond to the instructor’s feedback on their take-home exercises. The evaluation workshop will generally 
include peer discussion sessions, when students can share their learning and evaluation experiences. 


The grading scheme for self-evaluation (and the instructor’s adjustment) is as follows: 


e Grade A: Achieved all the learning goals relevant to the module 

e Grade B: Achieved almost all the learning goals (except for one or two evaluation criteria) relevant to the 
module 

e Grade C/Pass: Achieved most of the learning goals relevant to the module 


The grades A, B, and C may be qualified with ‘+’ or ‘—’, where applicable. The overall course grade will be the 
weighted average of the module grades as shown below (using the standard conversion A = 4.00, A— = 3.67, etc., 
according to http://www.tcnj.edu/~recreg/policies/grading.html#three, not 
http://www.tcnj.edu/~academic/policy/Grading.htm). 


Module A: 10% (lower weighting for practice purposes) 
Module B: 30% 
Module C: 30% 
Module D: 30% 


If a student far exceeds the standard, s/he may be assigned an A+. Although there will be no course grade beyond 
A, an A+ (for a module) could offset, say, A— in another module. For example, if a student receives A, A—, A, and 
A+ for the four modules, respectively, s/he will receive an A for the course. 


After almost every class meeting, there will be take-home exercises. Students are expected to do them and hand 
them in at the beginning of the next class meeting. Although these exercises will not be graded by the instructor, the 
instructor will give feedback on them and return them at the beginning of the next class meeting. When students 
self-evaluate, they will submit all the take-home exercises along with their responses to the instructors’ feedback. 
After submission of a module evaluation folder (called “portfolio’”), the instructor will adjust students’ self- 
evaluation, provide comments, and temporarily return them to students for review (when possible). Note that all the 
submitted work will eventually be kept with the instructor. 


Learning Activities 


The main learning activities consist of class meetings (two 80-minute sessions per week) and take-home exercises 
(expected amount of work equivalent to 360 minutes per week). Take-home exercises will be given after almost 
every class meeting and due at the beginning of the next class meeting (unless otherwise stated). Class meetings 
will contain components such as the following: 


Clarification of the learning goals as well as explanation of the evaluation form 

Survey (occasionally) 

Presentation of examples, cases, questions, and problems by students and the instructor 
Discussion of ideas, principles, and hypotheses known by the public and researchers 
Class and group discussion of different types 

Explanation and practice of using the evaluation form (esp. during Module A) 
Explanation of the take-home exercises given that day 

Evaluation workshop (at the end of each module) 


Course Topics (subject contents) 


1. Types of problems 
1. Practical, research, and computational problems 
2. Condition and cost/significance of a problem 
3. Set representation of a problem 
2. Notion of “theory” 
1. Applicability of a theory 
2. Definition in the context of mathematical logic 
3. Computability 
1. Formal models of computation, e.g., Turing machines (TMs) 
2. Decidability/undecidability including diagonalization and halting problem 
3. Reduction 
4. Church-Turing thesis 
4. Formal languages and automata 
1. Chomsky hierarchy 
2. Context-free languages (CFL), context-free grammars (CFG), push-down automata (PDA); properties of 
CFLs 
3. Regular languages, regular expressions (RegExp), finite automata (FA); properties of regular languages 
5. Complexity 
1. Tractability/Intractability 
2. Nondeterministic polynomial 
3. NP-complete 
4. Space complexity 
6. Interactivity 
1. Effects of interaction 
2. Formal models of interaction 
Textbook 


Hopcroft, John E., Motwani, Rajeev, and Ullman, Jeffrey D. 2001. Introduction to Automata Theory, 
Languages, and Computation, 2nd ed. Addison-Wesley. ISBN: 0-201-44124-1. [required (but listen to what 
the instructor will say about how we will use the textbook); available in the TCNJ bookstore; also available as 
3-hour reserve in the library] 


Schedule: Class meetings: Tue/Fri 4:00-5:20 p.m. in HH126 


Week| Date Unit Topic Exercises 
1 1/18 00 Introduction Your own computational problems 
1/21} Al |Problems: Anatomy and physiology Using a theory 
1/25| A2 |Theory: Form and content on ee about Theory ar 
2 Computation 
1/28} A3 |Theory of Computation; Possibilities and limitations Computing set membership 
3 2/1 | A4 |Turing Machines: Vehicle for mechanistic touring Test drive Turing machines 
2/4 | AS (Un)Decidability: When to play dice Module A Comprehensive Exercise 
4 2/8 | A6 [Module A Evaluation Workshop [eval form] Simulating a TM using a TM 
2/11, B1 Computability: Universal TM Universal TM review 
5 2/15, B2 Diagonalization: Magic of obliqueness Diagonalize other things 
2/18, B3 Halting Problem: Can we prove that we too will die? Infinite loop detection 
6 2/22| B4 Reduction: How to get more from reduced stuff Reduction; TM vs. Computers 
2/25| B5 Church-Turing thesis: Civil rights movement in its abstract form Module B Comprehensive Exercise 
3/1 | B6 [Module B Evaluation Workshop [eval form] AEM & wee powerrulonoe 
7 powerless? 
3/4 | B7 Computability review - 
- aad - |No class (Spring break) - 
g 3/15. Cl Chomsky hierarchy: Often, we want less power TM/grammar variants 
3/18 C2 CFG, CFL, PDA: Shopping-mall navigation CF examples 
9 3/22, C3 Regular expression, regular set, FA: Vending-machine operation Regular examples 
3/25 C4 Properties of regular sets: Pumping gas for free Pumping exercise 
3/29 C5 Properties of CFLs: Pumping air into the lung; Chomsky hierarchy Module C Comprehensive Exercise 
10 summary 
4/1 | C6 [Module C Evaluation Workshop [eval form] Data-size scalability 
Wl 4/5 | D1 |Complexity: Polynomial, Exponential, Nondeterminism Complexity analysis 
4/8 | D2 NP-complete: Why SAT rules Second civil rights movement 
2 4/12, D3 Space complexity: Is time the 4th dimension of space? Limitations with complexity analysis 
4/15, D4 Complexity review: What really matters TBA 
B 4/19 D5 “Real” complexity (1): Effects of interaction TBA 
4/22, D6 “Real” complexity (2): Modeling interaction Module D Comprehensive Exercise 
14 |4/26 D7 |D6: Final Evaluation Workshop [eval form]; Conclusion - 
Final |TBA ZZ Closing circle (optional; to be explained later) - 


e The schedule is tentative and subject to change. Please always refer to the on-line syllabus, which contains the 
latest information. 


e Course materials will be linked to the text in blue in the on-line syllabus. 


// End 


CSC460 (Spring 2005) Theory of Computation 


List of Definitions, Notations, and Concepts 


Jan. 14, 2005 


Note: The section and page numbers refer to the text (Hopcroft et al., 2001). For other references, see the reference 
list at the end of this document. 


Problems 


Practical problem [definition]: phenomenon (generally perceived negatively) that calls for some action 
[ref. Booth et al.] 
Research problems [definition]: question (yes-no or wh-question) that is generally crucial to solve a practical 
problem and calls for an answer [ref. Booth et al.] 
Computational problems [definition]: (informally) research problem that rends to computational some 
computational processing/analysis; (formally, within the traditional Theory) membership problem with respect 
to a certain set (thus, can be identified with the set); ~ language Sec 1.5.4 
Cost/significance (of a problem) [definition]: The “cost” of a problem is what one would suffer if the problem is not 
solved. The “significance” of a problem is what one would benefit if the problem is solved. Thus, these two 
notions are the opposite side of the same idea (i.e., you only need to identify one, not both). While the process 
of solving a problem would be objective and can be purely technical (“value-free’’), the cost/significance of the 
problem is subjective (“value-laden”). [ref. Booth et al.] 


Theo ry [ref. Enderton] 


Axioms [definition]: the very basic principles consistent with our experience; the starting point of a theory (Note that 
there is no way to “logically” justify these; these are assumed and generally obtained intuitively.) 

Rules of inference [definition]: the basic rules of obtaining additional information (statements) from the axioms 

Theorem [definition]: statement justifiable from the axioms and the rules of inference 

Proof [definition]: The “mechanical” process of justifying a theorem 

Logical consequence [definition]: If a statement follows another by analyzing their “meanings,” the former is called 
a “logical consequence” of the latter. (Note that the notion of “logical consequence” is stronger than that of “proof,” 
and thus, there may be a logical consequence of the axioms that may not be proven.) 

Theory [definition]: (informally) potentially infinite, consistent body of knowledge which can be systematically 
derived from a small number of abstract principles; (formally) the collection of all the statements that are 
logical consequences of the axioms 


Computability 


Turing machine (TM) [definition] Sec 8.2 
Effective procedure [description]: well-defined, step-by-step procedure Sec 8.2 
Algorithm [description]: effective procedure that always terminates ~ recursive Sec. 8.2.6, 9.2.1, p. 367 
(Computational) Problem [definition]: = set ~ property ~ predicate ~ characteristic function ~ language _— Sec 1.5.4 
Decidable [definition]: = recursive = TM-decidable = computable = solvable Sec. 1.1.3, 8.2.6, 9.2.1, p. 374 
Undecidable [definition] p. 302, 310, Ch 9 
TM-recognizable [definition]: = recursively enumerable (RE) Sec 8.2.5, p. 374 
Unsolvable [definition]: = non-RE = not TM-recognizable Sec 9.1, p. 374 


Semi-decidable [definition]: = undecidable (but not unsolvable) 
co-X (where X is some property) [definition]: The complement of the problem has property X. E.g., co-TM- 


recognizable. [ref. Sipser] 
Diagonalization [description] Sec 9.1 
Rice’s Theorem Sec 9.3.3 
Church-Turing thesis Sec 8.2.1 


Formal Languages and Automata 
Language [definition]: set (often, of strings) Sec 1.5.4 


Finite-state automaton (FSA; finite automaton, FA) [definition] Sec 2.1 


Deterministic finite-state automaton (DFA) [definition] Sec 2.2 

Nondeterministic finite-state automaton (NFA) [definition] Sec 2.3 
Regular expression [definition] Sec 3.1 
Regular language [definition] = regular set Sec 3.1 
Pumping lemma [description] Sec 4.1 
Context-free grammar (CFG) [definition] Sec 5.1 
Context-free language (CFL) [definition] Sec 5.1 

Deterministic context-free language (DCFL) [definition] Sec 6.4 
Push-down automaton (PDA) [definition] Sec 6.1 

Deterministic push-down automaton (DPDA) [definition] Sec 6.4 
Pumping lemma for CFLs [description] Sec 7.2 
Complexity 


Complexity (within Theory of Computation) [description]: the quantitative effects of the input size on 
computational performance 
Time complexity [description] p. 414 
Space complexity [description]: performance measure based on space (~ memory size) [ref. Sipser 1997] 
Worst-case analysis [description]: the performance analysis for the worst input pattern 
Average-case analysis [description]: the (estimated) average performance for all possible input patterns 
‘O’ (asymptotic upper bound) [description] [ref. Sipser 1997] 
Informal idea: Behaves roughly as bad as a certain function 
Definition #1: f(n) € O(g(n)) if there are constants c > 0 and mp 2 | such that f(n) < c g(n) for every integer 


n= No. ate f(n) 
Definition #2: f(n)e O(g (n)) if lim =c forsome c=0 
neo 8 n 
Class P (tractable) [definition] p. 361, Sec 10.1 
Class Exp (EXPTIME) [description]: O(c") for any constant c > 1 
Intractable [definition] p. 1, p. 361, Sec 10.1 
Class NP [definition] Sec 10.1 
Class NP-complete [definition] Sec 10.2 
Cook-Levin theorem [description] Sec 10.2.3 
Class NP-hard [definition] p. 423 


General 


Power set (of a set) [definition]: given set A, the set of all the subsets of A, i.e., {x |x CA} 

Deterministic [definition]: at most one transition/option for a unique situation 

Nondeterministic [definition]: possibly multiple transitions/options for a unique situation 

Reduction, reducibility [definition] Sec 8.1.3 
Polynomial time reducibility [definition] Sec 10.1.5 


References [all available in the library] 


Booth, Wayne C., Colomb, Gregory G., and Williams, Joseph M. 2003. The craft of research, 2nd ed. University of 
Chicago press. [excellent book on writing a research paper] 

Enderton, Herbert B. 2001. A mathematical introduction to logic, 2nd ed. San Diego, CA: Harcourt. [classic 
introduction; only the Ist edition (1972) is available in the library] 

Hopcroft, John E., Motwani, Rajeev, and Ullman, Jeffrey D. 2001. Introduction to Automata Theory, Languages, 
and Computation, 2nd ed. Addison-Wesley. 

Hopcroft, John E. and Ullman, Jeffrey D. 1979 Introduction to automata theory, languages, and computation. 
Addison-Wesley. [the first edition of our text; contains some more advanced materials] 

Sipser, Michael. 1997. Introduction to the theory of computation. PWS. [used as a textbook in the past] 


// End 


CSC460 References (Spring 2005) 
Apr. 20, 2005 
The books available in the TCNJ library are indicated with “L”. 


Standard texts in the Theory of Computation 

e  Greenlaw, Raymond and Hoover, H. James. 1998. Fundamentals of the Theory of Computation. Morgan 
Kaufmann. [another gentler introduction; little computability theory] 

e Hein, James L. 1996. Theory of Computation: An Introduction. Jones and Bartlett. 

e = Hopcroft, John E., Motwani, Rajeev, and Ullman, Jeffrey D. 2001. Introduction to Automata Theory, 
Languages, and Computation, 2nd ed. Addison-Wesley. [the textbook; L (3-hour reserve)] 

e Hopcroft, John E. and Ullman, Jeffrey D. 1979 Introduction to automata theory, languages, and computation. 
Addison-Wesley. [the first edition of our text; contains some more advanced materials; the discussion of the 
Chomsky hierarchy (Unit C1) is mainly based on this book; L (3-hour reserve)] 

e Lewis, Harry R. and Papadimitriou, Christos H. 1981. Elements of the Theory of Computation. Prentice-Hall. 

[another classic; L] 

e Linz, Peter. 2001. An Introduction to Formal Languages and Automata, 3rd ed. Jones & Bartlett. [very good, 
but little coverage of complexity theory; the discussion of the pumping lemma (Unit C3/C4) is mainly based on 
this book] 

e = =©Martin, John C. 1997. Introduction to Languages and the Theory of Computation, 2nd ed. WCB McGraw-Hill. 

[another undergrad text; L] 

e Sipser, Michael. 1997. Introduction to the theory of computation. PWS. [used as a textbook in the past; space 
complexity ideas (Unit D3) are mainly taken from this book; L (3-hour reserve)] 

e Sudkamp, Thomas A. 1997. Languages and Machines, 2nd ed. Addison Wesley. 


Other Theory of Computation references 

e = Atallah, Mikhail J, ed. 1999. Algorithms and Theory of Computation Handbook. CRC Press. [L] 

e —_ Baase, Sarah. 2000. Computer Algorithms, Introduction to Design and Analysis. Addison-Wesley. [classic text 
in algorithms; parallel algorithm examples (Unit D4) are mainly from this book] 

e Blum, Lenore, et al. 1998. Complexity and Real Computation. Springer. [L] 

e §=©Cai, Jin- Yi. 2003. Lectures in Computational Complexity. [available on-line: 
http://www.cs.wisc.edu/~jyc/8 10notes/book-ch1to5.pdf] 

e Chaitin, Gregory J. 1999. The Unknowable. Springer. [L] 

e Cutland, N. J. 1980. Computability: An Introduction to Recursive Function Theory. Cambridge Univ. Press. 
[good intro to recursive function theory; recursive functions and URM ideas (Unit B5/B7) are mainly from this 
book] 

e Davis, Martin. 1982. Computability and Unsolvability. Dover. [examples of unsolvable problems] 

e Denning, Peter J., et al. 1978. Machines, Languages, and Computation. Prentice-Hall. [L] 

e Dewdney, A. K. 1989. The Turing Omnibus: 61 Excursions in Computer Science. Computer Science Press. [a 
very readable introduction to various CS topics; the halting problem reading (Ex B1) and some Sample 
Problems are taken from this book; L] 

e Epstein, Richard L. and Carnielli, Walter A. 1989. Computability: Computable Functions, Logic, and the 
Foundations of Mathematics. Wardsworth & Brooks/Cole. [interesting collection of topics with a lot of 
excerpts] 

e Garey, Michael R. and Johnson, David S. 1979. Computers and Intractability: A Guide to the Theory of NP- 
Completeness. W. H. Freeman. [extensive collection of NP-complete problems; NPC problems and their 
connections (Unit D1/D2) are mainly taken from this book; L] 

e = Griffor, Er. R., ed. 1999. Handbook of Computability Theory. Elsevier. [L] 

e Harel, David and Rosner, Roni . 1992. Algorithmics: The Spirit of Computing, 2nd ed. Addison-Wesley. [a 

readable introduction to algorithms and computing; broader scope] 

Révész, Gyorgy E. 1991. Introduction to Formal Languages. Dover. [L] 

Rogers, Hartley , Jr. 1987. Theory of Recursive Functions and Effective Computability. MIT Press. [classic; L] 

Salomaa, Arto. 1985. Computation and Automata. Cambridge Univ. Press. [L] 

Traub, J. F. and Werschulz, A. G. 1998. Complexity and Information. Cambridge Univ. Press. [L] 


Logic and applied logic 


Ebbinghaus, Heinz-Dieter, Flum, Jéorg, and Thomas, Wolfgang. 1984. Mathematical logic. Springer-Verlag. 
Enderton, Herbert B. 2001. A mathematical introduction to logic, 2nd ed. Harcourt. [classic introduction; L 
(only the 1st edition, 1972)] 

Hintikka, Jaakko. 1996. The principles of mathematics revisited. Cambridge: Cambridge University Press. [a 
fresh, very critical role of FOL; discusses an alternative (still first-order) logic based on game-theoretic 
interpretation] 

Huth, Michael R. A. and Ryan, Mark D. 2000. Logic in Computer Science: Modeling and Reasoning about 
Systems. Cambridge Univ. Press. [intro to (semantic) model checking and (syntactic) program verification; L] 


Loeckx, Jacques, Ehrich, Hans-Dieter, and Wolf, Markus. 1996. Specification of abstract data types. Wiley. 
[emphasis on the logic-structure connection applied to software specification, lightly touched on in connection 
to the discussion of theories (Unit A2)] 


Beyond Turing computability 


Copeland, B. Jack. 2003. Hypercomputation. Minds and Machines 12(4):461-502. [very broad survey; an intro 
to the series of articles on hypercomputation] 


Copeland, B. Jack. 2000. Narrow Versus Wide Mechanisms: Including a Re-examination of Turing's Views on 
the Mind-Machine Use. Journal of Philosophy 97(1):5-32. [L] 

Copeland, B. Jack. 2000. Alan Turing's Forgotten Ideas in Computer Science. Scientific American 280(4):77- 
81. [L] 

Goldin, Dina. 2000. Persistent Turing Machines as a Model of Interactive Computation. In Foundations of 
information and knowledge systems: First International Symposium, FoIKS 2000, Burg, Germany, February, 
2000 (Lecture notes in computer science, 1762), eds. Klaus-Dieter Schewe and B. Thalheim, 116-135. Springer. 
[the idea of Interaction Machines (Unit D6) is taken from this article] 


Goldin, Dina and Keil, David. 2001. Interaction, Evolution, and Intelligence. In Proceedings of the Congress on 
Evolutionary Computation, Korea, May 2001. 


Israel, Navot and Goldenfeld, Nigel. 2004. Computational Irreducibility and the Predictability of Complex 
Physical Systems. Physical Review Letters 92(7):074105. [analysis of Wolfram’s ideas] 


MacLennan, B. J. 2003. Transcending Turing Computability. Minds and Machines 13(1):3-22. [used for Ex D4] 


Milner, Robin. 1993. Elements of Interaction (Turing Award Lecture). Communications of the ACM 36(1):78- 
89. [the idea of m-calculus (Unit D6) is taken from this article; L] 


Shagrir, Oron and Pitowsky, Itamar. 2003. Physical Hypercomputation and the Church-Turing Thesis. Minds 
and Machines 13(1):87-101. 


Siegelmann, Hava T. 2003. Neural and Super-Turing Computing. Minds and Machines 13(1):103-114. 
Siegelmann, HT. 1995. Computation Beyond the Turing Limit. Science 268:545-548. [L] 

Stannet, Mike. 2003. Computation and Hypercomputation. Minds and Machines 13(1):115-153. 
Steinhart, Eric. 2003. Supermachines and Superminds. Minds and Machines 13(1):155-186. 


Thomas, Wolfgang. 1990. Automata on Infinite Objects. In Handbook of theoretical computer science, ed. J. 
van Leeuwen, 133-191. Elsevier. 

van Leeuwen, Jan and Wiedermann, Jiri. 2001. The Turing machine paradigm in contemporary computing. In 
Mathematics Unlimited - 2001 and Beyond, eds. B. Enquist and W. Schmid, 1139-1155. Springer. [a draft 
version of (i.e., not necessarily identical to) the listed paper available at van Leeuwen’s web site: 
http://www.cs.uu.nl/people/jan/; used for Ex D5] 


von Neumann, John. 1967. The General and Logical Theory of Automata. In Cerebral Mechanisms in 
Behavior, ed. Lloyd A. Jeffress, 1-31. Hafner Publishing. 

Wegner, Peter. 1997. Why Interaction is More Powerful Than Algorithms. Communications of the ACM 
40(5):80-91. [L] 

Wegner, Peter and Goldin, Dina. 2003. Computation Beyond Turing Machines: Seeking appropriate methods to 
model computing and human thought. Communications of the ACM 46(4):100-102. [L] 


Wolfram, Stephen. 2002. A new kind of science. Wolfram Media. [extensive work on cellular automata; L] 


Not classified 


Booth, Wayne C., Colomb, Gregory G., and Williams, Joseph M. 2003. The craft of research, 2nd ed. 
University of Chicago press. [excellent book on writing a research paper; basis for the connection between 
practical and research problems as well as the reference to “cost”/”significance” along with a problem (Unit 
Al); L] 

Gottman, John M., Murray, James D., Swanson, Catherine C., Tyson, Rebecca, and Swanson, Kristin. 2002. 
The mathematics of marriage: dynamic nonlinear models. MIT Press. [L] 

Kline, A. David. 1998. Introduction (to Part 4 Theory and Observation). In Introductory readings in the 

philosophy of science, eds. E. D. Klemke, Robert Hollinger, and David Wéyss Rudge, 309-315. Prometheus 
Books. [some useful ideas about “theories” ] 

Michalewicz, Zbigniew and Fogel David B. 2000. How to Solve It: Modern Heuristics. Springer-Verlag. 
[discusses a variety of probably intractable (and other problems) from a practical point of view; L] 

Simon, Herbert Alexander. 1996. The sciences of the artificial, 3rd ed. MIT Press. [L (1st ed.)] 


e Stewart, David and Mickunas, Algis. 1990. Exploring phenomenology: a guide to the field and its literature, 
2nd ed. Ohio University Press. 

e Suppes, Patrick. 2002. Representation and invariance of scientific structures. CSLI Publications. [esp. on 
scientific theory and the logic-structure connection] 

// End 


CMSC485 (Section 2, Spring 2003) Theory of Computation Sample Problems 


This document contains sample problems for this semester. The problems are supposed to be 
‘practical’, at least to some extent. You may find some contexts where your action might be 
different depending on whether or not you know the answer (and why). If you are not interested 
in any of these problems or know how to solve these problems, there is no point for you to take 
this course. On the other hand, if you find some of these problems interesting and/or relevant, 
this course will introduce you to the Theory of Computation so that you will be able to answer 
not only these questions but also related ones, which might pop up in the future. The sample 
problems are collected from various sources. Some are adapted or re-created for this course. 


1. Infinite Loop Detection 


Infinite loop is one of the most common bugs in your (or anyone else’) program. It would be 
enormously helpful if someone writes a program to detect infinite loops in a given program (as a 
text file). Would it be possible to write such a program? 


2. Mayan Script 


You found an ancient Mayan script that is supposed to indicate the location of hidden treasures. 
The only language resources in that language you have access to are: (1) an extensive list of 
synonyms and (2) a collection of sentences whose meaning are already known. 


For example, let’s imagine that you need to translate the following sentence (obviously, not real 
Mayan): “koregawakarukane” with the following resources: 


Synonym list: “korega” = “maa”, “ruka” = “nnee’, “neene” = “darona’, etc. 
99 66 


Sentences whose meaning are known: “maawakanneedarona’”, “mosikasitarawakarukamone’’, 
etc. 


A sample session of translation by substituting synonyms would be (underlined words are 
replaced with italic words): 
“koregawakarukane” — “maawakarukane” — “maawakanneene” — “maawakanneedarona” 


Then, we can tell that “koregawakarukane” means the same thing as “maawakanneedarona”, 
which we can understand. If there are many sentences to analyze and the synonym list is 
extensive, we will naturally think of doing the task with a computer. Would it be possible to 
write a program to solve a problem of this kind? 


3. Floor Tiling 


Suppose that you are searching for a house to live in and have found a one you like, except that 
none of the floors are covered. So, you decided to buy the house any way and finish the entire 
floor with a variety of tiles available at the Home Depot. You will be using n types of square 
tiles with distinct patterns. To demonstrate your esthetic sense, you will match the edges of the 
tiles as shown below (in this case, using three types of tiles). [Images from Algorithmics by 
David Harel] 


(1) (2) 
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Note that if you choose a wrong set of tiles, you may never be able to demonstrate your esthetic 
capability, as shown below. 


(1) (2) (3) 
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Would it be possible to write a program to tell whether a given set of n types of square tiles can 
fill your floors with the above-mentioned condition? 


4. Linear Programming 

Consider the following problems: 

e Find the cheapest combination of foods that will satisfy all your nutritional requirements 

e Minimize the risk in your investment portfolio subject to achieving a certain return 

e In order to mass produce complex electronic circuit boards, an almost uncountable number of 


holes need to be drilled. How do we route the drill (attached to a robot arm) to visit all these 
holes so it takes the shortest possible route and time? 


All of these problem can be tackled by a technique called ‘linear programming’. Roughly, linear 
programming is an approach to set up k linear inequalities with n variables and find an 
assignment for the variables that satisfies all the inequalities. 


As many problems are rather complex, researchers have been solving them mainly through case- 
by-case approaches. Would there be a general way to solve all of these problems? 


5. Ambiguity Detection 


A simple rewriting system can be used to characterize a certain language (set of strings). For 
example, consider the following set of rewrite rules (can be used to represent programming 
language syntax): 


A->xAy (Rule 1) 
Arxy (Rule 2) 
A7>xxyy (Rule 3) 


Starting from the symbol A, we can generate all sorts of strings of the form x...xy...y where the 
number of x’s and that of y’s are the same. For example, applying Rule 1 twice and Rule 2 twice 
would result in an output ‘xxxyyy’ as shown below: 


ArxAyrxxAyyAxxxyyy 


One tricky point about this type of formulation is that the system can be ‘ambiguous’. That is, 
there may be more than one way of generating the same result. For example, the above system 
can generate ‘xxxyyy’ also by applying Rule | once and then Rule 3 once. Naturally, ambiguous 
systems could cause a lot of trouble for precise representation of a language and also for 
processing the language. Would there be a general way to detect whether this type of system is 
ambiguous? 


6. Program Verification 


One of the software industry’s biggest concern is how to check whether their products are really 
correct with respect to the specification that the developer and the user both agreed. According 
to a variety of sources, inability to do so will cost us an incredible amount of money in the future 
(well, this must be already happening). As you know, generality is a prime concern in Computer 
Science. So, why don’t we write a single program that could verify whether the program 
correctly solves a problem with respect to a given specification? But would it be possible? 


7. Arithmetic System 


One of the great achievements in logic is that it can formalize a wide variety of systems. For 
example, the mathematical system of arithmetic can be formalized in first-order logic in a 
consistent (roughly, meaningful) way. Its basic components include formulas such as “for any x, 
x+0=x” and “for any x, x x0 =0”. In this system, we can write all sorts of formulas that are 
representative of arithmetic. One property that a good logic system must have is that all ‘true’ 
formulas can be proven as a series simple steps. But here is a big question. Would it be really 
possible to prove all true formulas in the arithmetic system? 


8. Hilbert’s Tenth Problem 


A great mathematician Hilbert once asked: whether or not a given polynomial equation with 
integer coefficients has a solution in integers. Is this in general solvable? 


9. Program Elegance 


Many programmers may be trying to write ‘elegant’ programs (only if they have time, which is 
of course never available). But could we ever tell whether a particular program is elegant? 


10. Functional Programming Languages 


LISP/Scheme, ML, and Haskell are known as ‘functional’ programming languages. They differ 
from more popularized ‘procedural’ programming languages such as Basic and C. We can often 
observe that programs in a functional language look rather different from those in procedural 
languages, even for doing the same task. For example, you might never see a loop, only 
recursions. However, both of these language groups seem to be able to do pretty much the same 
thing. Could we justify that they can actually solve exactly the same class of computational 
problems? 


11. Cryptography 

Public key cryptography employs two types of keys, public and private. The basic idea behind 
the use of two keys is that (A) the public key can easily be obtained from the private key and (B) 
the private key cannot easily be obtained from the public key. For example, RSA cryptosystem 
satisfies these conditions. However, cryptology is a never ending battle. We will need to have 
many other cryptosystems for coming years (in fact, also right now). Then, how can we draw the 
line between ‘easy’ and ‘not-easy’ computation in a systematic manner? 


12. Monkey Puzzle 


The following is a snapshot of a game to place 9 square cards so that pictures will match (note 
that the cards cannot be rotated). [Image from Algorithmics by David Harel] 


In general, we can think of a puzzle in any n Xn size. How fast could we solve such a puzzle? 


13. Digital Circuits 


To design digital logic circuits, we need to analyze the relation between inputs and the output. 
For example, the following formula represents a circuit with four inputs and one output, each of 
which could take either 0 or | (‘+’ for OR, ‘-’ for AND, and ‘” for NOT). 


output = (i' + Ip + 13 ao i4) ‘ (i,' + Ip + 13 + iq') . (iy' + in' + 13 + iq') 


Given an arbitrary formula, we can surely find out whether some combination of inputs would 
result in output 1, by checking all the combinations (you should be able to tell how long it would 
take to compute all the results for the n-variable case). However, we would surely hope that 
there is a better way. How fast could we compute the output in general? 


14. Professor Assignment 


It is not a simple task of assigning all the CS faculty members to all the courses offered by the 
CS department. Well, it is actually not that complicated at TCNJ. But what about a large 
English department in a giant state university? Imagine that as a part of your work study, you 
are asked to write a program to do the scheduling for a large department. How fast in general 
could your program schedule all the faculty members? 


15. Knapsack Problem 


There will be an excursion tomorrow. You will need to pack all of your junk food packages in 
your knapsack. However, you realized that not all of them would fit in the knapsack. So, you 
want to fit as many as possible while the total cost of your junk foods is maximized (to impress 
your pals). How fast could you find out the solution given an arbitrary scenario? 


16. Cross-Country Interviews 


Suppose that you are invited for interviews (graduate schools or industry jobs) in a number of 
cities in the US. Unfortunately, they do not pay for your travel. So, you will need to minimize 
the expense. How fast in general could you find the best way to travel all the cities? 


17. CPU Register Allocation 


CPU is no doubt the ‘brain’ of a computer. We want to cram as many things as possible in 
CPU’s. However, we also need to consider the cost of doing so. In practice, we have to settle 
down at some cost-performance level. The cost-performance trade off also applies to the number 
of registers in a CPU. In order to find the optimal number of registers for a future CPU, we want 
to write a program to find the minimal number of registers for a collection of popular programs. 
What would be the performance of such a program? 


18. Map Coloring 


It has been shown that with four distinct colors, we can color any map so that the neighboring 
countries (or whatever political boundaries) do not share the same color. With three colors, we 
may or may not be able to do the same thing. How fast in general could we find out whether a 
map can be 3-colored? 


19. Time-Space Tradeoffs 


If our computers had an infinite amount of memory, we could load all the programs and run them 
without accessing the hard disk. This would eliminate our frustration with loading time. 
However, this will remain as dream. Then, we need to compromise the use of space (memory, 
etc.) and the processing time. Can we say anything general about the time-space tradeoff? 


20. Respectively in English 


In English, you can say something like: a, b, c, and d are the lower case of A, B, C, and D, 
respectively. In general, there is no limit to the number of items that are connected in this 
manner. What would be the simplest mechanism that can process the correspondence of 
multiple sequences like this? 


21. Programming Language Parsing #1 


Modern programming languages allow an arbitrary level of nesting. You do not need to write a 
LISP program to be baffled by an insane number of ‘(’ and ‘)’ as well as all sorts of other nesting 
constructions. What would be the simplest representation (i.e., that cannot represent anything 
more complicated) for this type of conditions? What would be the simplest mechanism that can 
handle the ‘nesting’ property of modern programming languages? In addition, are there 
properties of programming languages that would require more than handling nesting? 


22. Programming Language Parsing #2 


Suppose that we know the answer to the previous problem. In other words, we know how to 
characterize the set of all valid programs in terms of program structure. Then, given an arbitrary 
program, can we always identify whether it belongs to the set? In other words, is parsing 
possible? Of course all sorts of programming languages are being parsed by all sorts of 
compilers and interpreters. The question here is more general because we are talking about all 
programming languages with, e.g., the nesting property, including those we have never seen. 


23. Password Screening 


When you obtained an account for a host computer, you might have been asked to choose a 
password that satisfies the conditions: (i) at least six characters, (i1) must include at least one 
symbol or numeral, (iii) must include at least one upper case character, etc. It shouldn’t be 
difficult to check all these conditions. But in order to maintain the security of the system, this 
kind of screening must definitely be done by a program. What would be the simplest 
representation (i.e., that cannot represent anything more complicated) for this type of conditions? 
What would be the simplest mechanism that can handle this and similar conditions in a 
systematic manner? 


<End> 


CSC460 (Spring 2005) Module A Evaluation Form 


Name Self-evaluation (A, B, C possibly with +/-) 
Adjustment by the instructor 


Evaluation Materials (Portfolio) 

Your evaluation materials (referred to as “portfolio,” and to be placed in the provided manila folder) consist of the following 

Items: 

1. This form (must be filled out; see the instructions below) 

2. Word-processed supporting notes responding to the instructions in this form (except for the materials completed during the 
evaluation workshop) 

3. Take-home exercises (including the comprehensive exercise), chronologically ordered 

4. Materials completed during the evaluation workshop (to be explained in class) 


Self-Evaluation Procedure 

During the module, before the evaluation workshop 

e You regularly examine the learning goals check list (included below). 

e If you think you satisfy a criterion, (i) place a check mark in the box (M) at the end of the criterion, and (ii) in your 
supporting notes, explain how you satisfied the criterion in a way the instructor can be convinced. That is, you are expected 
to explain the process, not just the consequence. 

e If you believe that you already wrote your response to a certain criterion in an earlier exercise, you can simply refer to that 
exercise (which must be included in your portfolio). 


During the evaluation workshop (preliminary) 

e At the beginning of the evaluation workshop, you must have Items 1 through 3 as hard copies, as well as blank sheets for 
Item 4. 

e There will be an in-class, open-ended module review exercise to check your understanding of the learning goals. Although 
your answers will not be graded per se, they may be used to validate your supporting notes. For example, if you thought you 
achieved the learning goals and could still not be able to respond to the review exercise well, you will need to improve your 
self-evaluation (Performance Goal 7). 

e You will have an opportunity to share your portfolio with other students. 

e Finally, you will write a reflective essay revealing your thought process during the evaluation workshop, and assign yourself 
a grade based on the course grading scheme (included at the end of this sheet). 

e At the end of the session, you will submit your portfolio. 


After the evaluation workshop 

e If your written justification for a criterion is convincing, the instructor will also place a check mark next to yours. 
e If necessary, the instructor will adjust your grade. 

e Normally, your portfolio will be returned in the next class meeting. 


Learning Goals Checklist (the goals not pursued in this module are “ ”) 
In your supporting notes, clearly identify the criteria, e.g., Cla (for Content Goal 1 Criterion a), P5b (for Performance Goal 5 
Criterion b), referring to the labels below. 


Content Goals 

1. Practical problems can often be transformed into research and computational problems. Every problem is 
associated with cost/significance, which is relative to the evaluator. Computational problems can be 
represented as a set, readily available as the input to computational processing. [problem] 
a. Understood the connections and differences among practical, research, and computational problems, as 


well as the notion of cost/significance. O 
b. Explained why a research problem can be transformed intoaset.. O 
c. [optional; if necessary] Reviewed concepts in discrete math (sets, relations, functions, mathematical 

structures, logic; ref. http://www.tcnj.edu/~komagata/cmsc210/03f/Topics.pdf). (QO) 


2. A theory is a potentially infinite, consistent body of knowledge which can be systematically derived from a 
small number of abstract principles. The gap between the principles and the entire information is the source of 
the theory’s predictive power. Also being abstract, a theory can be applied to a broad range of phenomena, 
which might appear distinct. [theory] 


4. 


a. Understood the notion of theory, referring to its “predictability” and “applicability,” using your own 


EX AILS ES a, cert ta eafaece Pate cette wh at TRS iN Ne Ot AN a el A OB oN ND Ds dom an Kod hoe O 
b. Explained how Theory of Computation could impact your career involving computation. O 
The algorithmic notion of computation can be represented in a variety of equivalent forms, which define a 
bounded class of sets. That is, there is a limit to algorithmic computation. [computability ] 
a. Understood the basic mechanism of Turing machine, partly through the use of asimulator, O 
b. Explained the connection between computational problems represented as sets and the use of Turing 

machines to process them. O 
c. Explained following the notions using a schematic diagram: decidability, undecidability, TM- 

recognizability, unsolvability, semi-decidability, O 


Performance Goals 


1. 


2: 


Identify real-world problems which are relevant to the student’s life and can be tackled by computational 

means.[awareness] [combined with other goals] 

Transform real-world problems into research problems, and then into computational problems, along with the 

analysis of the cost/significance of a problem. [transformation] 

a. Understood the process through exercises. [simply refer to successfully-completed exercises] O 

Analyze computational problems with respect to , computability, , and 

[analysis/evaluation] 

a. Analyzed the computability (e.g., decidability) of a variety of problems including your own, using Turing 
machine as a model of computation. O 


Express ideas orally and in writing, in a manner clearly understood by other students (with equivalent 
background). Explain your own ideas orally and in writing, clearly and logically. Revise the ideas, reflecting 
the feedback from other students and the instructor. [communication] 


a. Completed all the exercises (take-home and in-class), O 
b. Responded to the other students’ and the instructor’s comments (e.g., on your exercises). O 
Take initiative in both independent and group activities. 

[initiative] 
a. Regularly contributed to class and group discussions/activities. O 
b. Regularly examined the evaluation criteria and placed check marks on this evaluationform. O 


Reflect upon the student’s own thinking process and assess the student’s own performance relative to the 

content and performance goals. [reflection] 

a. Was able to reflect upon your experience in this module through the activities during the evaluation 
WOPKSNOP iE). ce. te:5.<82.3.0toof¥e.ntfo4 bin Steet state elele ts ies cuese. tha lvicY He geteseteus tt ote e3ut [during the eval workshop] 

b. Self-evaluated your achievements accurately. [during the eval workshop] 


Self-Evaluation Criteria 
At the end of the module evaluation workshop, propose your grade based on the following scheme (possible qualification with 


+/-): 


Grade A: Achieved all the learning goals relevant to the module 
Grade B: Achieved almost all the learning goals (except for one or two evaluation criteria) relevant to the module 
Grade C/Pass: Achieved most of the learning goals relevant to the module 


// End 


CSC460 (Spring 2005) Module B Evaluation Form 


Name Self-evaluation (A, B, C possibly with +/—) 


Adjustment by the instructor 


Evaluation Materials (Portfolio) 
Your evaluation materials (referred to as “portfolio,” and to be placed in the provided manila folder) consist of the following 


Items: 

1. This form (must be filled out; see the instructions below) 

2. Word-processed supporting notes responding to the instructions in this form (except for the materials completed during the 
evaluation workshop) 

3. Take-home exercises (including the comprehensive exercise), chronologically ordered 

4. Materials completed during the evaluation workshop (to be explained in class) 

Learning Goals Checklist (the goals not pursued in this module are “21 ”) 


In your supporting notes, clearly identify the criteria, e.g., Cla (for Content Goal 1 Criterion a), P5b (for Performance Goal 5 
Criterion b), referring to the labels below. 


Content Goals 
3. Interactive computation subsumes algorithmic computation, but not vice versa. That is, there is a qualitative 
difference between these two modes of computation. [interactive computation] 
a. Was able to point out the limitations of the traditional, algorithmic approach to “computability.” O 
4. The algorithmic notion of computation can be represented in a variety of equivalent forms, which define a 
bounded class of sets. That is, there is a limit to algorithmic computation. [computability] 
a. Understood how to create a universal TM, including how to represent a TM as the input on the tape. O 
b. Understood when and how to use the diagonalization technique. O 
c. Can explain logically that the halting problem is semi-decidable and infinite-loop detection is unselyable 
non-TM-recognizable. O 
d. Understood how to use “reduction” to (i) establish equivalence among models and (ii) analyze properties 
applied to different problems. O 
e. Understood the description and the significance of Church-Turing thesis. O 
fi Canexplain (i.e., teach) this goal to CS students outside this class. O 
7. Power set, also encompassing the distinction between determinism and nondeterminism, can introduce 


discontinuity with respect to computability and complexity. [power set] 
a. Understood (i) that the power set is always “larger” than the original set and (ii) why the power set of a 
countable set is uncountable. oO 


Performance Goals 


Respect, analyze, and give constructive criticisms to the ideas in the literature and those expressed by other 

people. [critical attitude] 

a. Critically analyzed the following points: (i) the use of the infinite tape in TM [You may assume that every 
concrete object is finite.], (ii) the diagonalization technique, (iii) no tolerance to errors as the standard 
position in the Theory of Computation, (iv) the lack of interaction in TM, (v) any other aspects O 

b. Critically analyzed the usefulness of the “computability” area of the (traditional) Theory of Computation O 

c. Critically analyzed the course materials/organization esp. in connection to how you could learn most 
(=i cre] Bhi, =| | a a See Ss A no Ee ee O 

Express ideas orally and in writing, in a manner clearly understood by other students (with equivalent 

background). Explain your own ideas orally and in writing, clearly and logically. Revise the ideas, reflecting 

the feedback from other students and the instructor. [communication] 

a. Completed all the exercises on time (take-home and in-class). 

b. Responded to the other students’ and the instructor’s comments (e.g., on your exercises). 

Take initiative in both independent and group activities. Also extend the domain of theoretical inquiry beyond 

the scope prepared by the instructor. [initiative] 

a. Chose a research question for the mini research project and started to explore it. 

b. Noted any other aspects relevant to this goal. 

Reflect upon the student’s own thinking process and assess the student’s own performance relative to the 

content and performance goals. [reflection] 

a. Was able to reflect upon your experience in this module through the activities during the evaluation 
WOLK SHOP xe sons tascote a soectes cana aaa he et tera ae a ean tan rt aanect aac oMcestsast [during the eval workshop] 

b. Self-evaluated your achievements accurately. [during the eval workshop] 


Self-Evaluation Criteria 
At the end of the module evaluation workshop, propose your grade based on the following scheme (possible qualification with 


+/-): 


Grade A: Achieved all the learning goals relevant to the module 
Grade B: Achieved almost all the learning goals (except for one or two evaluation criteria) relevant to the module 
Grade C/Pass: Achieved most of the learning goals relevant to the module 


// End 


CSC460 (Spring 2005) Module C Evaluation Form 


Name 


Self-evaluation (A, B, C possibly with +/-) 


Adjustment by the instructor 


Evaluation Materials (Portfolio) 
Your evaluation materials (referred to as “portfolio,” and to be placed in the provided manila folder) consist of the following 


Items: 


1. This form (must be filled out; see the instructions below) 

2. Word-processed supporting notes responding to the instructions in this form (except for the materials completed during the 
evaluation workshop) 

3. Take-home exercises (including the comprehensive exercise), chronologically ordered 

4. Materials completed during the evaluation workshop (to be explained in class) 


Learning Goals Checklist (the goals not pursued in this module are “ ”) 
In your supporting notes, clearly identify the criteria, e.g., Cla (for Content Goal 1 Criterion a), P5b (for Performance Goal 5 
Criterion b), referring to the labels below. 


Content Goals 


5. The computable class of sets contain a hierarchy of proper subsets which can be characterized by distinct 
grammars and automata. [formal languages and automata] 
Understood that there is a hierarchy of TM-recognizable languages that can be specified and processed by 


a. 


b. 


the corresponding grammars and automata, respectively [Chomsky hierarchy]. 


Understood that TM-recognizable languages can be specified by unrestricted grammars (rewriting system), 


which are equivalent to TMs. Also understood why this class is not useful in practice. 
Understood that context-free languages (CFLs) can be specified by context-free grammars (CFGs) and 
processed by push-down automata (PDAs). Also understood when/how to use this class to analyze 
problems. 
Understood that the deterministic subset of CFLs is an important class that supports the backbone of 
programming languages, which can be processed by deterministic PDAs (DPDAs). 
Understood that regular languages (regular sets) can be specified by regular expressions (RegExps) and 
processed by finite-state automata (FSAs). Also understood when/how to use this class to analyze 
problems. 
Understood how to use the pumping lemma to show that a language is notregular. 
Understood how to use the pumping lemma for CFLs to show that a language is not context-free. Also 
understood that certain properties of CFLs are undecidable. 
Could explain (i.e., teach) this goal to CS students outside this class. 


7. Power set, also encompassing the distinction between determinism and nondeterminism, can introduce 
discontinuity with respect to computability and complexity. [power set] 


a. 
b. 


Understood that nondeterminism can be represented as the power set of the possible states. 
Understood why nondeterminism affects the power of PDAs but neither TMs nor FSAs. 


OO oO 


OO 


Performance Goals 


4. Respect, analyze, and give constructive criticisms to the ideas in the literature and those expressed by other 
people. [critical attitude] 
a. Critically analyzed the usefulness of the “languages/automata” area of the (traditional) Theory of 
Computation, esp. in connection to your ability to choose the minimal specification/process for the given 
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5. Express ideas orally and in writing, in a manner clearly understood by other students (with equivalent 
background). Explain your own ideas orally and in writing, clearly and logically. Revise the ideas, reflecting 
the feedback from other students and the instructor. [communication] 
a. Completed all the exercises on time (take-home and in-class). O 
b. Made conscious efforts to promote transfer of learning among students, esp. during in-class exercises (e.g., 
explained what you understood to other students and learned what you didn’t understand from other 
iL 14 [2111 6S) oe RE ee I OE Re RN eee a ee ed say O 
6. Take initiative in both independent and group activities. Also extend the domain of theoretical inquiry beyond 
the scope prepared by the instructor. [initiative] 
a. Continued to analyze your own problem and mini research project, by applying the notion of “modularity” 
and identifying the simplest mechanisms for the component modules. oO 


b. Noted any other aspects relevant to this goal. O 
7. Reflect upon the student’s own thinking process and assess the student’s own performance relative to the 
content and performance goals. [reflection] 
a. Was able to reflect upon your experience in this module through the activities during the evaluation 
WOEKSNODP vse 3: saesctecee es ceearam Nae Sees teeta se aktea chan dies Note tae [during the eval workshop] 


b. Self-evaluated your achievements accurately. [during the eval workshop] 


Self-Evaluation Criteria 

At the end of the module evaluation workshop, propose your grade based on the following scheme (possible qualification with 
+/-): 

e Grade A: Achieved all the learning goals relevant to the module 

e Grade B: Achieved almost all the learning goals (except for one or two evaluation criteria) relevant to the module 

e Grade C/Pass: Achieved most of the learning goals relevant to the module 


// End 


CSC460 (Spring 2005) Module D Evaluation Form 


Name 


Self-evaluation (A, B, C possibly with +/—) 
Adjustment by the instructor 


Evaluation Materials (Portfolio) 
Your evaluation materials (referred to as “portfolio,” and to be placed in the provided manila folder) consist of the following 


Items: 


1. This form (must be filled out; see the instructions below) 

2. Word-processed supporting notes responding to the instructions in this form (except for the materials completed during the 
evaluation workshop) 

3. Take-home exercises (including the comprehensive exercise), chronologically ordered 

4. Materials completed during the evaluation workshop (to be explained in class) 


Learning Goals Checklist (the goals not pursued in this module are “21 ”) 
In your supporting notes, clearly identify the criteria, e.g., Cla (for Content Goal 1 Criterion a), P5b (for Performance Goal 5 
Criterion b), referring to the labels below. 


Content Goals 


3. Interactive computation subsumes algorithmic computation, but not vice versa. That is, there is a qualitative 
difference between these two modes of computation. [interactive computation] 


a. 


b. 
Cc. 


Understood the effects (and limitations) of parallel computation with respect to the three subareas of the 
traditional Theory of Computation (in a sense as a preliminary to interactivity), O 
Understood what kind of problems cannot be adequately represented by TMs. O 
Understood the basics of super-Turing computation (more “powerful” than TMs) including its significance. 


Was able to speculate where the theoretical underpinning of computer science should be heading, in order 
to offer robust analyses of a variety of computational problems. oO 


6. The practicality of an algorithm depends on its complexity relative to the input data size. [complexity] 


a. 


b. 


Reviewed how to interpret the big O notation [game-theoretically and using the on-line graphing tool 
(http://www.tcnj.edu/~komagata/Graphing)].. 
Understood that exponential growth with respect to the input data size (time complexity) is considered 
impractical (“intractable”), through examples. O 
Understood the class of “nondeterministic polynomial” (NP) problems, through examples. Also 
understood (i) why there are so many NP problems and (ii) why NP problems are essential for computer 
security. O 


Understood the notion of “polynomial time reducibility” and its impact on relating problems. O 


Understood the class of “nondeterministic polynomial complete” (NPC) problems, through examples. Also 
understood (i) the basics of how to show that a problem is in NPC and (ii) why this class is important. O 


Understood the essential difference between time and space complexity, as well as the hierarchy involving 
various time/space complexity classes. O 


Could explain (i.e., teach) this goal to CS students outside this class. O 


7. Power set, also encompassing the distinction between determinism and nondeterminism, can introduce 
discontinuity with respect to computability and complexity. [power set] 
a. Understood that power set introduces exponential growth, which could lead to intractability. 
b. Gained insight into the meaning of power set (in contrast to just knowing the formal definition), which is 
reflected in all three subareas of the traditional Theory of Computation (and also in reality). O 


Performance Goals 


4. Respect, analyze, and give constructive criticisms to the ideas in the literature and those expressed by other 
people. [critical attitude] 
a. Critically analyzed the usefulness of the “complexity” area of the (traditional) Theory of Computation, i.e., 
Content: Goalie: (6. ee en een coe ec hn, pe setae Se ttn ese Seneca Bete a ta Baers Sl ap oes te pM te ah O 
b. Critically analyzed other students’ practicum presentation, esp. in connection to aspects relevant to the 
Theory of Computation. == [to be assessed by the instructor on 4/27/05; no need to write] 
5. Express ideas orally and in writing, in a manner clearly understood by other students (with equivalent 
background). Explain your own ideas orally and in writing, clearly and logically. Revise the ideas, reflecting 
the feedback from other students and the instructor. [communication] 
a. Completed all the exercises on time (take-home and in-class). In particular, clearly articulated the 
understanding of the Theory of Computation in the mini research paper [Comprehensive Exercise]. O 
6. Take initiative in both independent and group activities. Also extend the domain of theoretical inquiry beyond 
the scope prepared by the instructor. [initiative] 
a. Continued to analyze your own problem and mini research project, by applying the notion of “complexity,” 
162, Content) GOaliGs <a o0<t * sat gosoet coy oes 0 No I GOS EEE ak tatoo nus ones ats. genet! O 
b. Asymbol (form) can mean different things (content) to different people, depending on the context [ref. 
Unit A2]. An analogous idea applies to a course as well. For example, this course could mean different 
things to different people; a positive attitude might lead to a positive outcome. What kind of initiative did 
you take to make your experience in this course positive? O 
7. Reflect upon the student’s own thinking process and assess the student’s own performance relative to the 
content and performance goals. [reflection] 
a. Was able to reflect upon your experience in this module through the activities during the evaluation 
WOEKSHOP et 22, c: 2 2r.fo. 5 Stuitts oth, Hesate.cteicteliteleieler lets bie ctaleial feietelene Bietete sets [during the eval workshop] 


b. Self-evaluated your achievements accurately. [during the eval workshop] 


Self-Evaluation Criteria 

At the end of the module evaluation workshop, propose your grade based on the following scheme (possible qualification with 
+/-): 

e Grade A: Achieved all the learning goals relevant to the module 

e Grade B: Achieved almost all the learning goals (except for one or two evaluation criteria) relevant to the module 

e Grade C/Pass: Achieved most of the learning goals relevant to the module 


// End 
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Today 
Understand the course learning goals 
Understand the course organization 
— Syllabus 
— On-line resources 
Identify the first things to do 
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Section 1 
Course Learning Goals 
Content goals 
— Understand ... (ideas) 


Performance goals 
— Being able to do ... (actions) 
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Preview: Theory of Computation 


impossibility Computability 


Uninteractable? 


Interactable? 
x Ss. 
distributivity? | Undecidable 
= 
j Decidable 
real practicality - NS 
Intractable Context-free 


Complexity) Tractable Regular 

Complexity) 1"? ie. 
practicality Languages) simplicity 
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This Course 


* Theme: Apply the Theory of Computation 
— Application: Your computational needs 
— Theory: A small number of principles 


Approach: Problems, Transform, 
Compute, Discuss, and Evaluate 

— Emphasis on discussion during class 
— Take-home exercise after every class 
— No memorization; no exams 


CSC460 00 Initial survey: General, Technical 


Content Goals: Understand 


. Problem 

. Theory 

. Interactive computation 

. Computability 

. Formal languages and automata 


. Complexity 
. Power set 
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Performance Goals: Do 


. Awareness 

. Transformation 

. Analysis/evaluation 
. Critical attitude 

. Communication 

. Initiative 

. Reflection 
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Section 2 


Course Organization Course Modules 


Syllabus . Problems, Theory, Computability (1) 
On-line resources . Computability (2) 


— Course web pages . Formal Languages and Automata 
— Instructor's page for students . Complexity 
— SOCS: E-mail, discussion, grades 


Textbook 
Schedule, Course modules | 
Alignment of goals - assessment - activities 
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Assessment Class Meetings 


* Module evaluation materials (portfolio) Clarification of the learning goals, etc. 
. Evaluation form Survey (occasionally) 
- Support notes (word-processed) Presentation of examples, cases, etc. 
. Take-home exercises (and in-class notes) Discussion of ideas, principles, etc 
. Materials completed during Evaluation i , 
Class and group discussions 


Workshop 
Carry these materials to class Practice using the evaluation form 


Explanation of the take-home exercises 
ea Evaluation workshop 
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Current Plan and Flexibility Self-Introduction 


* The current plan based on my experience * Name 
— Observed some positive aspects in other ¢ How you want to be called 
courses 
* This class: A small number of presumably 
motivated students 
— Can modify the plan fairly flexibly 


— Your suggestions on course organization 
welcome 


« Why you are here (i.e., taking this course 
or was at least interested in this course) 


CSC460 00 CSC460 00 


Section 3 


First Things to Do 


By the next class meeting on Friday 
* Exercise 00 [all exercises to be put on-line] 
— Part 1 (essay): Your own problem(s) that can 


be solved computationally Class discussion next time. 


— Part 2: Course Preview 
* Read and understand the syllabus 
* Read Module A Evaluation Form 
+ Visit the course page [try most links] 
— Read the instructor’s page for students 


* Set SOCS discussion e-mail notification 
CSC460 00 
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Name: 


Exercise 00, 1/18/05 


Note: All take-home exercises are due at the beginning of the following class meeting (unless 
otherwise stated). For this exercise, the due date is 1/21/05, this coming Friday. At that time, 


submit this sheet with other required components. 


Most of the take-home exercises in this course will be open-ended. This reflects the belief of the 
instructor that “important” real-world problems are often open-ended (in other words, problems 
with well-defined answers are not so important), and that students must practice facing such 
problems in regular courses as well as other opportunities such as research and internship. If you 
feel more comfortable with being able to present the “expected” answers to well-defined 
problems, these exercises will be opportunities to develop a different attitude to problems. Such 
an attitude would actually be more useful when you begin a career after graduation, be it in the 
industry or graduate work. 


Part 1: Your Own Problem(s) 


Probably the best way to learn and appreciate the power of Theory of Computation is to apply it 
to computational problems that you care. In addition, it would be more effective if we discuss 
issues that you are familiar with and/or interested in. Thus, it is extremely important that you 
think and write about problems you know at this stage. 


Task: Write an essay about one or more of your own problem(s) that can be solved 
computationally (i.e., using a computer or defining a computational process such as algorithm). 
Try to be reasonably detailed so that the reader can understand your problem(s) well. 
Optionally, you may want to analyze certain computational aspects (e.g., whether computable 
within a reasonable amount of time) as much as you can. Be flexible and creative. Look around 
and think carefully. In addition, be prepared to discuss your problem in class. 


Instructions/Notes: 

1. Word-process your essay and submit a hard copy at the designated time. oO 

2. Include: basic course info, exercise ID (00 for this one), your name, and date. oO 

De ADO MOC INCMIE aS Pasee eo os eos. Succes aac tas neat vaice, vale: eaatts cae. nuecs dtnvele gil a nase ase case ceuele oO 

4. Try to be concise (no extraneous information) and clear (understandable by other college 
CT oe fe. 94.8 34, tae et desde a hee De PT he eee naan | 

5. Include page numbers, if more than one page. eee, oO 


6. No requirements on font selection/size, line space, margin, or the number of pages. Apply 
your common sense. 


Part 2: Course Preview 


Just to make sure that you understand the course organization and content well, do all of the 
following things (sorry to be too prescriptive; you will have more flexibility as we go): 


Tasks: 
1. Read the syllabus. 
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a. If you understand it completely, check the box at right. 

b. If you have questions, check the box and list them on the other side of this sheet. 
2. Read Module A Evaluation form. 

a. If you understand the general idea about evaluation, check the box at right. 

b. If you have questions, check the box and list them on the other side of this sheet. 
3. Visit the course page and most of the links including the instructor’s page for students. 

a. If you understand the “Course work” section of it, check the box at right. 

b. If you have questions, check the box and list them on the other side of this sheet. 
4. Log on to SOCS and set discussion board e-mail notification. 


Survey: Time spent between classes for this course (this exercise, etc.): 


// End 
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OOoOogagdago 


What to do with your problems? 
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Exercise 00 Part 1 


* Your own problems 
« Analysis of computational aspects 
(optional) 
— Can obtain an answer? 
— Can do it with available resources? 
— Can do it within a reasonable time? 
— Can handle different cases of the problem? 


CSC460 At Why problems? 


Research Problems 


* As questions 

— Yes-no question: “Is Furby male?” 

— Wh-question: “What is the sex of Furby?” 
* Cost or significance 


— Suffering of not being able to answer or 
benefit of answering [opposite sides] 
— Often, in connection to a practical problem 
* Example: “How do ants find food?” 
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Unit A1: Overview 


Discuss your own problems 


Analyze the process of going from 
problems to solutions 


Identify different types of problems 
Discuss how to represent problems 
Practice transforming problems 
Preview Exercise A1 “Using a Theory” 
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Problems 


Problem transformation 
— From more realistic to more manageable 
Problem types 


— Practical problems: Calls for an action (in 
reality) 


— Research problems: Calls for information 


— Computational problems: Calls for 
computation 
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Digression: Analogy 


Problem (question) and significance 
Course (materials) and significance 
Organization and role 

Anatomy and physiology 


CSC460 At Feeding problems to a computer 
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Preliminary 


Computational Problems Imagine a Lazy Machine 


¢ Research problem 1: “Is Furby male?” * Only responds with “uh” or “nah” 


— Computational problem (a la Prolog) * Question: Can we phrase various 


+ Input: furbySex (male) . Paranal problems in a way even the lazy machine 
* Output: e.g., no. can still be useful? If so, how? 


* Research problem 2: “What is the sex of 


— Computational problem (a la Prolog) 


* Input: furbySex (X) . 
* Output: e.g.,X = female. 
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Converting Questions Group Exercise 1 


* Any question (including wh-questions) can . 
be converted into a series of yes-no 
questions. 
— Wh-question: “What is the sex of Furby?” 
— Yes-no questions: “Is Furby male?” “Is Furby 
female?” 
¢ This technique allows us to deal with 
complex problems with “lazy” machines. 


Convert the following wh-questions into a 
series of yes-no questions 

1. What are the planets (of the Sun)? 

2. Where did you eat? 

3. When will the last human be born? 
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cSC460 At Representing formally? 10 


Formal Representation Number of Arguments 


* wh: “What is the square of x?” Function * 2 arguments 


-f={d, 1), (2, 4, (3, 9), ...} —E.g., “Is 1 the square of 1?” “Is 4 the square of 
—le., f()=1,f(2)=4, fF) =9, .. 2?” etc. 


— Pairs of an input and the output — Cf. “What is the square of what?” 

* yes-no: “Is y the square of x?” ‘Relation: — {, 1), 2, 4), 3, 9), } = {@ 9) |y = 27} 
-R={(, b, 2, 4), G, 9), 3} * 1 argument 
-lLe, (DER (24 ER, 3, IER... —E.g., “Is 1 a square?” “Is 4 a square?” etc. 
— Pairs of two inputs — Cf. “What are squares?” 


; — — {1, 4, 9, ...} = {x | xis a square of some number} 
bacaeaet Discrete math: review if necessary ee 


12 


Computational Problem 


* Checking the membership of a set (could 
be a relation) 
* Then, we may just identify a computational 
problem with the set. ~ language Recall this? 
* Examples (cf. Group Exercise 1) 
— {Mercury, Venus, Earth, Mars, Jupiter, ... } 
— {(Nobo, Penang), (Furby, home), ...} 
— {2005, 2006, 2007, 2008, 2009, 2010, ...} 


cscasoar Potential problems with this description? 


Practice: “Dating” 


* Practical problem: Date an ideal partner. 
« Research problem: 

— Cost/significance 
* Computational problem (set): 
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Group Exercise 3 


* Identify each of your own problem (Ex 00) 
with respect to the following stages: 
— Practical 
— Computational 
— Set 


* Transform your problems so that you 


eventually convert them into sets (may 
need to make adjustments) 
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Interim Summary: Problem Transformation 


Practical problem ~ action 

— Significance: Real-life situation 

Research problem ~ question 

— Significance: possibility of solving the 
associated practical problem (if acted) 

Computational problem ~ set 


— Significance: Abstract/computational 
treatment > Only yes-no response needed 
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Group Exercise 2 


Transform the following research 
problems into the corresponding 


computational problems, and represent 
them as sets 


1. Is it going to rain this evening? 
2. Why do people (still) smoke cigarette? 
3. How can one cook pasta? 
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Unit Summary 


¢ Types of problems: practical, research, 
computational 


— Why categorize? 

* Computational problem ~ set 
— Why set? 

¢ Problem transformation 
— How to do it? 
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Exercise A1: Using a Theory 


* Part 1: Using a Theory 
— Task 1: Identify theories (in/out CS) 
— Task 2: Define “theory” (your own) 
— Task 3: Personal feelings about theory 


* Part 2: Review “Problems” 
— Group Exercise 3 (previous slide) 
¢ Unit A1 Summary question 
— Were you sufficiently motivated to discuss 
“oroblems” as the starting point of this 


course? Explain. 
CSC460 A1 
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Name: 


Exercise Al, 1/21/05 


Part 1: Using a Theory 


This is a course about Theory behind many computing concepts. However, the notion of 
“theory” is not necessarily well understood or appreciated. Before we can discuss components 
of the Theory of Computation, we had better confirm our understanding of theory in general. 


Task 1: Identify multiple examples of theory (both in and outside computer science). Then, 
discuss how they can be used in practical situations, and comment on their usefulness. 


Task 2: Give your own definition of “theory.” Do not look up any reference. Be flexible and 
creative. 


Task 3: Describe your personal feelings about theory. Were you always comfortable and 
content? Were there any time you felt that some theory is too abstract and/or useless? 


Be prepared to discuss this part in class. 


Part 2: Review “Problems’’ 


A lot of our activities (computational or not) begin with a problem. Thus, a good grasp of 
problems at different levels for appropriate processing is an important skill. 


Task: Concisely write up your response to Unit Al Group Exercise 3 (in class; slides available 
on-line). That is, re-do the exercise for your own problem from Exercise 00. Certain problems 
may not be neatly analyzed in the way discussed in class. If your problem falls in that category, 
explain why. 


Note: If you think you understood how to do this part, you should place a check mark for the 
criteria in Content Goal | (eval form). Then, you can fill in your supporting notes with a brief 
description of how you were able to achieve those criteria. If you have questions, contact the 
instructor right away. 


Instructions/Notes (for both parts): 

1. Follow the general guidelines given in Exercise 00. 
2. Clearly identity parts and Tasks i000 ht el ea ols tale le el ee he he 
3. Review the evaluation form and start to write your supporting notes. 


Survey: Time spent between classes: 


// End 
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Before Starting Unit A2: Overview 


* Questions about Ex A1 Part 2 (Review) Discuss your experience with “theory” 
* A practical problem of the past weekend: Explore notions of theory 
show shoveling Discuss important properties of theory 


—=Hesearcluproblen? Preview Exercise A2 “Your Ideas about 
— Computational problem (set)? Theory of Computation” 


* Why theory? Use of your problems in this course 
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Exercise Ai Part 1 Theory 


* Part 1: Using a Theory * Consistent, i.e., no contradiction 
- Task 1: Identify theories (in/out CS) * Potentially infinite amount ~ predictability 
— Task 2: Define “theory” (your own) + Principled, i.e., derivable from a small 


— Task 3: Personal feelings about theory number of principles via reasoning 


Non-theory 
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Optional 
Theory (in Logic) Theory, More Formally 


é Consistent i e. no contradiction + Language: Specification of the use of symbols. 


— Formula (well-formed formula): Syntactically correct sequence of symbols that 


+ Potentially infinite amount ~ predictability taints sabne 


+ Interpretation: Specification of the meaning of symbols under consideration. 


e Axiomatized i e derivable from a small + Structure: Specification of the meaning of symbols not defined by the 
: as ay 


interpretation. 
number of axioms via rules of inference + Satisfaction: A structure A satisfies a formula 9 if 9 is true with respect to A. 
This relation is usually written as A |= 9. 
+ Logic: Combined specifications of language, interpretation, and satisfaction. 
+ Model: For a logic L, a structure A is called a model of formulas ©, if A |= @ for 
all@ e &. The collection of models of ® is written as Mod(®). 


A Non-theor + Logical consequence: For a logic L, a formula 9 is called a logical 
Rules of inference Oneneory consequence of 9, if for each A e Mod(®), A |= 9. 


+ Theory: For a logic L, a theory is a set of formulas ® such that for each 
formula ¢, ® |= 9 implies @ ¢ ®. [In practice, we also require consistency.] 


Are These Theories Useful? 


Example 1 

— Axioms: Newton's law of physics 

— Rule of inference: Scientific reasoning 
— Theory: Classical physics 

Example 2 

— Axiom: “0” is a number. 


— Rule of inference: If nis a number, so is n+ 1. 


— Theory: The set of natural numbers 
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Digression: Analogy 


Theory and significance 
Logic and (mathematical) structure 
Syntax (sentence organization) and 
semantics (meaning) 
Form and content 
Recall Unit A1 
+ Problem (question) and significance 
* Course (materials) and significance 


* Organization and function 
+ Anatomy and physiology 
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Caveat: Axioms 


* For a theory to be useful, axioms must be 
“appropriate” with respect to our 
experience and/or intuition. 

* However, axioms are still assumptions. In 


general, there is no way to logically justify 
their correctness. 
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Significance of a Theory 


* The notion of “theory” (at least ina 
technical sense) simply identifies a way 
some knowledge is organized. 

The usefulness of a theory depends on 
factors outside the theory. For example, 
the axioms and rules of inference must 
“reflect” our observation and intuition. 

— Cf. problem and its cost 
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Practice: “Good” Theory 


* Discuss the following criteria proposed to identify 
a good theory, with respect to our 
characterization of theory. 


1. Reflect the real world? 

2. Supported by convincing evidence? 

3. Explain the past and predict future outcomes? 

4. Handle new data and discoveries? 

5. Clearly understandable, simplify rather than 
complicate the world? 
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Interim Summary: Theory 


Form 

— Axioms, rules of inference, theory 
Content 

— Grounding (reflecting observation/intuition) 
— Significance (usefulness) 

— Nature of axioms 


— Properties: consistency, predictability, 
efficiency, etc. 
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Practice: Blood Vessels 


Schematic representations 
Comparison with other systems 
Involved properties 


Developing a theory for analogous 
phenomena 

— Axioms: 

— Rule of inference: 

— Theory: 
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Word Freq FxR Word Rank | FxR 
the 3332 3332 turned 200 | 10200 
and 2972 5944 you'll 300 | 9000 
and 1775 5325 name 400 | 8400 


he 877 8770 comes 500 | 8000 


410 8200 group 600 | 7800 
294 8820 lead 700 | 7700 
222 8880 friends 800 | 8000 
8600 begin 900 | 8100 
9480 family 8000 
9660 brushed 8000 
9920 sins 6000 
10440 Could 8000 


10400 Applausive 8000 
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Graphical Representation (2) 


10000 72 


+ 
100 1000 10000 
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Preparation for Group Exercise 1 


Anything in Common? 


Word frequency (in a text) 
City size 

Intensity of wars 

Web page links 

Forest fires 

Earthquakes 

Heartbeat 
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oo + + 
2000 4000 6000 8000 
Rank 
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Power Law 


For all the mentioned phenomena 


* (roughly) The scale-frequency relation as 
a straight line with the slope —1 on a log- 
log scale graph. 


* (formally) Freg « Scale =! 
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Web Site Visits Mail List Activities 


Number of list members 
Activity of Poland-L and APAP mailing lists members 
vanuary 1997 - June 2000 


APAP 
Ay = 100 /x 2% 
Poland-L * 


y=70/x°? * 


Number of postings 


1000 
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Material Fracture Pattern Shore Erosion 


Run 2, Coefficient of Variation of erosion rate 


) ~~ vs separation time 
€ 
S 
3 + 
+ 
B : 


| 


herd 
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“Pleasant” Musical Tones Group Exercise 1 
yy 6U6SlUe ¢ Examine and evaluate the following theory 


(with respect to consistency, predictability, 
significance, etc.) 

— Axioms: Many real-world phenomena (natural 
and artificial) “scale” (i.e., small < large). 

— Rule of inference: If a phenomenon scales x 
times, such a phenomenon is x times unlikely 
to occur. 

— Theory: Many real-world phenomena follow 
the “power law” (Freq « Scale ~'). 
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Group Exercise 2 Group Exercise 3 


Develop a theory of “justice” 


¢ What kind of theory would be useful to 
— Axioms: 


analyze/solve your own problems? 
— Rules of inference: Speculate the form of the theory (axioms, 
— Theory: rules of inference, theory). 
Examine the following properties 
— Consistency 
— Predictability 
— Usefulness 
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Examine the following properties 
— Consistency 

— Predictability 

— Usefulness 


Preview 


Unit Summary Problems and Theory 


Theory: Axioms, rules of inference, theory 
Significance of a theory 

Nature of axioms Research problem Research problem 
Analyzing theory : aon 


Practical problem| |Practical problem Pranicel problem 
Re 


SN Process involved between these? 


Process involved between these? 


ue 
Computational problem (set) 


+ What can we do after this? 
+ Where do theories apply? 
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Exercise A2: Theory of Computation 


Your ideas about Theory of Computation 


— Task 1 (review): Identify/analyze CS “theories” you 
know 


— Task 2 (~ Group Exercise 3): Usefulness of the 
theories you know; missing elements 


* Unit A2 Summary questions 


— Do you have a better idea about “theory” after 
this meeting? Explain. 


— Questions/Comments/Suggestions 
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Name: 


Exercise A2, 1/25/05 


Your Ideas about Theory of Computation 


In this course, you are expected to improve and refine your understanding of the Theory of 
Computation so that you can apply it to your problems that would surface in your career (life). 
In this part of the exercise, you will examine your current understanding related to the Theory of 
Computation, as a starting point. Note that you are not expected to “study” to respond to this 
exercise. Write as much as you know at this point. 


Task 1 (review): Identify ideas/concepts you know (in Computer Science) that can be considered 
as theories. Try to analyze both the organization (1.e., axioms, rules of inference, and theory) 
and the significance of each of the examples. Although many ideas discussed in CS may not 
have been labeled as “theories,” there are many concepts that can be considered theory. If you 
have difficulty coming up with a candidate, identify some theory (not necessarily in CS, e.g., 
Unit A2 Group Exercise 2 “Justice’”’) and do the same. 


Task 2 (in connection to Unit A2 Group Exercise 3): Examine whether any of the theories you 
identified in Task 1 is useful for analyzing/solving your own problems (e.g., those in Exercise 
00). If you think the theories you know are not sufficient to tackle your problems, what kind of 
theory would you need? 


Be prepared to discuss this part in class. 


Instructions/Notes: 
1. Follow the general guidelines given in earlier exercises. cece eee erect eect, O 


Survey: Time spent between classes: 


// End 
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Unit A3 Supplement, 1/29/05 


Here is my response to your summary questions (rephrased/combined in some case). 


Q1: 


Al: 


Q2: 
A2: 


Qs: 


A3: 


Q4: 


Ada: 


Possible to express the “actual” solution of a problem in set notation, esp. when the number 
of possible solution is very large? How general the predicate notation on a set could be. 
The list notation is good only for a finite set. But as long as the set is finite, you can still 
contain a large number of members, at least theoretically. Many problems have infinitely 
many solutions. The “problem” (or solution) set for such problems must be in the predicate 
notation. If the specification part (at the right of the bar ‘|’) of a representation is precise 
enough for the “processor” to interpret, the predicate notation is good enough. For 
example, {(c, e) | c causes e} would be good only for informal discussion, not for a 
computational process, while {(x, y) | xX < y, where x, y € NaturalNumbers} can be 
processed mechanically. 


Representation of sets besides the basics? 

When you deal with a real-world problem, the data can be complex. As an example, let us 
consider Group Exercise 3 (Slide 24), which we did not have time to discuss. Here is the 
first attempt: {(course, instructor) | instructor teaches course}. Would this be right? No, 
because it characterizes a single instance of course-instructor mapping. The problem must 
address the entire map that we get the complete information about who is teaching what in 
within realistic constraints. How about this: {(courses, instructors, assignment) | 
assignment is an injective function from courses to instructors, where courses and 
instructors are non-empty sets of available courses and instructors}? For example, imagine 
courses = {410, 460, 470}, instructors = {NN, NK, PD, MM}, and assignment = {(410, 
PD), (460, MM), (470, NN)}. assignment is a function (why?) and injective (why?). Now, 
you should be able to see how the materials discussed in Discrete Structures begin to play a 
role. [Review questions: What’s wrong if assignment is not a function or injective? Why 
doesn’t assignment need to be surjective? ] 


Problems in CS for which it is impossible to prove whether they are “computable” or not? 
How to finally decide if something is “computable” or not? 

To answer these questions, we need to know a little more about the notion of 
“computability,” which we will discuss in the rest of Module A and much of Module B. 
Keep your question. As a side note, I want to mention a logical problem that cannot be 
proven true or false. Consider the sentence: “I’m telling a lie.” Am I really telling a lie? 
This “liar paradox” has a close connection to “computability.” We will probably touch on 
this point later. 


The distinction between what is necessarily included in the discussion about complexity vs. 
what is excluded? 

The complexity area of the Theory of Computation focuses on the time or space 
performance of algorithms as a function of their input size. In this area, we only discuss 
algorithms, which must terminate for all inputs; thus, computability is not an issue. In 
addition, we only deal with mechanisms that are capable of solving the problem in hand; 
thus, choosing the simplest mechanism is not an issue. 
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Q5: 


AS: 


Q6: 
A6: 


How to apply/figure out the properties in the three subareas of the Theory in “my” 
problems or other problems like the floor tiling? 

The entire course is supposed to be the opportunity for you to gain this ability. So, at this 
point, you should not worry about not being able to do this as much as you wish. However, 
I hope you got some sense of the three subareas of the Theory and be able to guess which 
areas are more relevant to your own problems. To get some additional ideas, you may 
want to use the sample problems from Spring 2003 
(http://www.tcnj.edu/~komagata/csc460/05s/SampleProblems.pdf). Problems 1 through 10 
primarily involve issues in computability; Problems 11 through 19 primarily involve issues 
in complexity; Problems 20 through 23 primarily involve issues in languages/automata. As 
I mentioned in class, most of your problems have some relevance to some of these areas. 

In addition, if your problem calls for a program that should not terminate or involves high 
degree of interactivity, it might require analysis beyond the traditional Theory. For 
example, the robotic control of automobile must not terminate and my Birds program is 
most appropriately designed as interaction of autonomous birds. 


How can formal models of interactivity be mathematically expressed? 

I am planning to include a very basic discussion related to this topic near the end of the 

semester. Some references are listed in the section, “Beyond Turing computability” of the 

references file (http://www.tcnj.edu/~komagata/csc460/05s/References.pdf). Here is a 

preview of the discussion. To address interactivity (or computation beyond algorithms), 

people have proposed different approaches: 

O Instead of discrete data/processing, use analog computation (e.g., analog version of 
artificial neural network) 

oO Instead of limiting to a single input-processing-output session, allow the mechanism to 
repeat the process with some memory between sessions 

oO Instead of prohibiting interaction within the process, allow the process to interact with 
(or consult) another process. 

With any of these approaches, we can show that it would become possible to do what was 

not possible with a single session of algorithmic computation. 


If you have more questions, let me know. 


// End 
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Before Starting 


* Questions 
* Computational problems as sets 
— Need to know the answer in advance? 


— Representing the relation between inputs and 
the problem schematically? 
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Exercise A1 Part 1 


* Your ideas about Theory of Computation 


— Task 1 (review): Identify/analyze CS 
“theories” you know 


— Task 2 (~ Group Exercise 3): Usefulness of 
the theories you know; missing elements 
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Problems and Theory 


Practical problem| [Practical problem| (Practical problem 
Ls 


\ "foe . 
‘\_ Identifying relevant questions 


Research problem 


Research problem 


Representing computationally (set) 


Computational problem (set) 


+ Possible to compute at all? 
+ How to choose the simplest mechanism? Theory 
* Practical for any future inputs? 


CSC460 A3 Significance? 


5 


Unit A3: Overview 


Discuss your ideas about “Theory of 
Computation” 


Preview the main components of the 
Theory of Computation 


Practice representing computational 
problems as sets 


Preview Exercise A3 “Computational 
problem solving” 
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Problems and Theory 


Practical problem| |Practical problem Prapicel problem 
Ne 


Z 
Z 


The + Identifying 


Research problem 


Research problem 


+ Representing computationally (set) 


Computational problem (set) 


Properties to analyze (about the problem)? 
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Preview: Theory of Computation 


impossibility |Computability 
Uncharted area 


Uninteractable? 


Interactable? 
E. = 


Traditional area 
Distributivity? Undecidable 
SS 
Decidable 
ra — 
Intractable Context-free 


Complexity) Tractable Regular 


practicality Languages’ simplicity 
Automata 
CSC460 A3 6 


real practicality 


Example 1: Floor Tiling Computability 
* Content Goal 4 
* Example questions 


— What are the limitations of computation? All 
the computational problems (sets) solvable? 


¥ ‘ — How can we compare different forms of 
? “computation?” 


— What is the notion of “computation?” 
Images from Algorithmics by David Harel © Sign ificance? 


« Are your own problems “computable?” 
CSC460 A3 
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Example 2: Survivor 


Scenario 


Languages/Automata 


* Content Goal 5 
— Alone on a large island where you survived an + Example questions 


aircraft crash. — Where are the balance between the simplicity of 
— Established a base where you can spend your computational mechanisms and their abilities? 
nights safely — How can we identify an appropriate computational 
, mechanism for a given problem (set). 
— Still need to explore the island to obtain foods. 


Significance? 
* Appropriate (minimal) mechanism for your The simplest mechanism for your own 
“mental” computer? problems? 
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Example 3: Scheduling 


* How to define the problem? 

¢ Must be sufficiently general to be able to 
handle cases including: 
— Your time management computation? 
— Organizing a team of workers — What would be the limit of “practical” 
— Assigning courses to instructors computation? 


— Schedule the analysis process for the entire * Significance? 
human genome 


Complexity 


* Content Goal 6 
¢ Example questions 
— How does the input data size affect the 


* Practicality of your own problems, esp. 
facing large data? 
CSC460 A3 
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What is not usually discussed... 


* Content Goal 3: Interactivity 
* Example questions 


— How well does algorithmic computation fare in 


real-world computation? 


— What are the essential properties that are 
needed to solve real-world problems 
computationally? 


* Significance? Examples? Implications? 


cscisona These questions as computational problems? 


Interim Summary 


* Theory of Computation 


— Offers principles in three main aspects: computability, 


languages/automata, complexity, most commonly 
dealing with problems as sets 


— Limitations due to its traditional foundation on 
algorithmic computation, cf. interactivity 
* Caveat 


— Focus on computational problems (sets), i-e., no 


systematic methods for problem transformation, cf. 
discrete math 
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Group Exercise 1 


Suppose that a typical compilation 
problem has been transformed into a 
computational problem (set). 

Analyze with respect to the following: 

— Computability (possibility) 

— Languages/Automata (simplest mechanism) 


— Complexity (practicality with respect to the 
input size) 


CSC460 A3 


Preview: Theory of Computation 


impossibility |Computability 
Uncharted area t 


Uninteractable? 


Interactable? 
== 


Traditional area 
Distributivity? Undecidable 
es 


er Decidable 
| practicality - aa 


intractable Context-free 
Complexity Tractable 


Regular 

x a 

practicality Languages’ simplicity 
Automata 
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Practice: Robotics 


* Suppose that robotic control of an 
automobile has been transformed into a 
computational problem (set). 

* Analyze with respect to the following: 

— Computability (possibility) 
— Languages/Automata (simplest mechanism) 


— Complexity (practicality with respect to the 
input size) 
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Set Representation of Problems 


¢ Benefits 


— To analyze/compute such problems, we only 


need a single mechanism to check set 
membership 


— Well-developed techniques in mathematics 
are available 


¢ Limitations? 
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Alternative Forms 


Computational problem 

Set 

Language 

— Set of strings (special case of set) 
Characteristic function 


— Outputs yes/no, i.e., comparable to a relation 
(set of tuples) 
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Set Notation (Review) 


List notation (technically, only for finite sets) 

— {1, 2, 3, 4, 5} 

— {(paper, stone), (scissors, paper), (stone, scissors) } 

Predicate notation 

— {x|0<x <5, xis a natural number} 

— {(x, y) | x wins over y in the paper-scissors-stone 
game} 


— {x |x is a number never used by the human} 
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Group Exercise 2 


¢ Infinite loop is one of the most common 
bugs in any program. It would be 
enormously helpful if someone writes a 
program to detect infinite loops in a given 
program. Give the set representation of 
the computational problem involved here. 
Speculate the basic Theory properties 
(i.e., possibility, etc.), referring to the set (a 
concise, informal description suffices). 
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Set Examples 


1. What is the sex of Furby? 
{female} 
2. What are palindromes? 
{a, cc, wow, abba, kayak, hannah, ...} 
3. Addition operation (on natural numbers) 
{(0, 0, 0), (0, 1, 1), (1, 0, 1), C1, 1, 2), ...} 
4. Causal relation (on any possible event) 
{ (drink, joy), (drink, nausea), (drink, addiction), ...} 


CSC460 A3 Not knowing answers; Avoiding “...” 20 


Practice: Predicate Notation 


* Palindromes 
{a, cc, wow, abba, kayak, hannah, ...} 
= {x |x is a palindrome} “Abetter way? 
={ } 
¢ Addition operation (on natural numbers) 
{(0, 0, 0), (0, 1, 1), A, 0, 1), C1, 1, 2), ...} 
={ } 
* Causal relation (on any possible event) 
{(drink, joy), (drink, nausea), (drink, addiction), ...} 
} 
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Group Exercise 3 


* For any (possibly very large) academic 
department, how can we assign all the 
courses to the instructors within the usual 
constraints (e.g., a single instructor for a 
single course)? Give the set 
representation of the computational 
problem involved here. Speculate the 
basic Theory properties, referring to the 
set. 
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Unit Summary 


Theory of Computation: computablity, 
languages/automata, complexity 


Set representation of computational problems: 
list/predicate notations 


Understand Unit A3 Exercise 
Summary question 


— We discussed broad ideas with little details. 
So, you must have questions or be uncertain 
on at least some aspect. What are they? 
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Name: 


Exercise A3, 1/28/05 


Part 1: Computational Problem Solving 


As we glanced at the main components of the Theory of Computation, we will be ready to 
discuss the components more in detail. The standard interface between your practical problems 
and the theoretical analysis is the set representation of your (corresponding) computational 
problem. 


Task 1 (Program Verification): One of the software industry’s biggest concern is how to check 
whether their products are really correct with respect to the specification that the developer and 
the user both agreed. According to a variety of sources, inability to do so will cost us an 
incredible amount of money in the future (well, this must be already happening). As you know, 
generality is a prime concern in Computer Science. So, why don’t we write a single program 
that could verify whether the program correctly solves a problem with respect to a given 
specification? Give the set representation of the computational problem involved here. 
Speculate the basic Theory properties (i.e., possibility to solve computationally, the simplest 
mechanism, practicality with large data), referring to the set (a concise, informal description 
suffices). Can you think of such a program? 


Task 2 (Map Coloring): It has been shown that with four distinct colors, we can color any map 
so that the neighboring countries (or whatever political boundaries) do not share the same color. 
With three colors, we may or may not be able to do the same thing. Give the set representation 
of the computational problem involved here. Speculate the basic Theory properties, referring to 
the set. 


Part 2: Review “Theory of Computation” 


Without going into the details of the Theory of Computation, we discussed the properties 
associated with the three traditional subareas of the Theory. While it must be difficult to point 
out exactly how these would apply to your problems, you should still be able to speculate how 
these property would apply (intuition is important!). 


Task: Suppose that your own problem (Ex 00) has been transformed into a computational 
problem (set). Concisely analyze the problem with respect to the basic Theory properties. If you 
have difficulty, explain where you have the difficulty. You are also encouraged to discuss with 
other students and/or the instructor. 


Part 3: Evaluation Form and Supporting Notes 

Module A evaluation will come in a week or so. You must be continuing to fill in the evaluation 
form and writing up your supporting notes as much as you can. You are also encouraged (but 
not required; i.e., not required to submit anything for this part) to attach a copy of your 
supporting notes to this exercise so that the instructor can comment on them. 


Survey: Time spent between classes: 


// End 
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Questions/Review Unit A4: Overview 


Ex A3 


— Part 1: Computational Problem Solving — Intuition first; reasoning later (Module B) 
* Task 1: Program verification 


Introduce Turing machines (TMs) 
* Task 2: Map coloring — : ' 
; — Definition, properties, behaviors 

— Part 2: Review 

; , . Learn how to operate TMs 
— Part 3: Evaluation form and supporting notes Preview E ise Ad “Test-Drive TMs” 
Schematic representation pesos aka } Se ee (Ms 

; — Using the JFLAP simulator 

— Set representation of a problem 
— Analyzing the set regarding Theory properties 
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Overview: “Computability” 
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Overview: Theory of Computation Tools for Computability Analysis 


impossibility Computability 


¢ Need for a standard abstract mechanism 
Uninteractable? of computing sets 


Focus in Modules A & B * Some possibilities 
Welene lee 


— — Some modern programming language 
ee : 

Distributivity? eeamsnned — Some assembly language 

Eiselettals — Some kind of pseudo code 


intractable Context-free — Mathematical functions 
Complexity) !ractable Regular ~ Other 


real practicality 


~ 
practicality Languages 
Automata 
CSC460 A4 


simplicity 
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Turing Machine (TM) Example Processes 
* Proposed by Alan Turing (1936) 


¢ Word-by-word translation from English to 
Spanish, e.g., “I want Taco Bell” 

Machine with a finite number of states ¢ Detect palindromes 

Some mechanism to control 


¢ Detect binary numbers 
State 0 | State 1 | State 2 State k 


Machine with a finite number of states 
Some mechanism to control 


State 0|State 1|State 2}- State k 
nfOjo 


blank symbols 
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Additional Properties TM Input 
: ss tape has no ged sh _ + Any string (of a designated alphabet) 
7 iti itten. ; : 
_ ee a pee essere * One instance of the complete information 


— A tape position may be blank ((). The positions to be tested for its acceptability (cf. yes-no) 
outside the input are filled with blanks. Example 


— The initial tape position is the leftmost input position. 
sa — For the problem {(x, y, z) | x+y =z, where x, y, z 
* There are finite number of states. : . 
F : € NaturalNumbers}, the following can be given 
— There is a unique start state. F a ” 
: as an input “...41#2#31...”. A reasonable TM 
— There are designated accept states. 


Id t this input. Cf. the binar 
* The input/tape alphabet contains a finite number fone cae Be pre Hes y 
of symbols. Sar bee 
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Possible TM Behaviors TM’s Yes-No Response 


Halts * Option 1 (ternary) 
—Accepits: If in one of the accept states — Yes: If accepts (of course, halting) 
—Dies: Otherwise — No: If dies (still halting) 
LOOPS Note: Assume no loping rom an accepting slate alee aa 
* Option 2 (binary) 
Information left on the tape — Yes: If accepts cel ae malting) 
: : — Not yes: Otherwise (i.e., no distinction 
— Often, ignored (side effects) : ‘ 
‘ : . between dying vs. looping) 
— Possible to interpret it as the output 
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TM Variations TMs vs. Computers 


Single accept state 

Both accept/reject states 
Semi-infinite tape 

—l.e., the existence of the left end 
Multiple tapes 
Nondeterministic transition 
—l.e., 5 as arelation, not a function 
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¢ Similarities and differences? 
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Interim Summary 


* TMs are a simple, intuitive, and precise 
model of computation. 

* The behavior of a TM is to capture the 
notion of “computation,” also referred to as 
“effective procedure.” 

¢ When an effective procedure is 
guaranteed to terminate, it is called 
“algorithm.” 


CSC460 A4 TM Practice 13 


Formal Definition 


TM M=(Q,%,T, 5, q, 4, F) 
* @Q: finite set of states RERRCateal 
¢ LZ: finite set of input symbols, 4 ¢ © 
I: finite set of tape symbols, Ue T, 2cT 
5: transition function Ox > OxI x {Left, Right} 
— E.g., {(q); 9), (q 1, 8), (a, D, (Gy, 8, D), 3 
qo: the start state g, € O 
4: blank symbol (B in the text) 
F: set of accept states F< Q 
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“Language” Examples 


« Any sequence of 0’s and 1’s with exactly 
one ‘#: {a#y |x, y € {0, 1}*} 

« Any sequence of 0’s and 1’s repeated after 
exactly one ‘#’: {w#w|we {0, 1}*} 

« Any distinct sequences of 0’s and 1’s 
delimited by ‘#’: {#x,#x,...#x, | each x, € {0, 
1}* and x; 4 x, for each i #j} 

« Any sequence of the same number of 0’s 
and 1’s in that order: {071" | n > 0} 
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Transition Diagram (JFLAP) 


Transition 

* Read ‘1’ 

* Replace it with ‘4’ 
¢ Move Right 


-@) 


Accept state 


Head movement on the tape 
ie) ‘e) ce 
AUErtH tere saxsaeesaich [eeeceduscea 


Step: 0 1 2 
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Practice: Detecting Input 


¢ Draw a transition diagram of a TM that 


would accept when there is some input 


(limit to 0 or 1). Fe {2)—'="--G) 


* Formally define the TM.» 


TM M=(0,3,T,8,49,0,F) 
+ @Q: finite set of states 
+ 2: finite set of input symbols, 5 ¢ 
+ T: finite set of tape symbols, [¢T,2cT 
+ 6: transition function OQ xl > OxT x {Left, Right} 
* qo: the start state g, ¢ O 
+ [: blank symbol (B in the text) 
CSC460 A4 + F:set of accept states Fc O 


Unit Summary 


Understand the basics of Turing machines 
— Description, properties, behaviors 
— Transition diagram, formal definition, JFLAP 


Start Exercise A4 “Test-Drive TMs” in groups, in 
the lab 


Summary question 


— lf TMs did not exist, what would you use as a 
model of computation? Explain. 


— Questions/Comments/Suggestions 
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Unit A4: Another Binary Adder Example, 2/6/05 


After reviewing your TM binary adders, I came up with my own version. The main idea is 
similar to Jared’s multiple-tape TM (but without using multiple tapes). I also borrowed Eric’s 
idea of using the little endian format (but with the input formatting following the exercise sheet, 
Le., 1+01=11 for 1+2=3, without using additional tape symbols for “carry’’). If one still wants to 
get input in the big endian, it would be possible to add a preprocessor to convert the format, 
using Scott’s binary number reverser. 


Here is the explanation of the main mechanism. The TM would compare the leftmost bit of each 
segment. In the following example, there is nothing wrong with the part “O + 1 = 1”: 


.. Off +fi=fo100... 


If it were “O + 1 = 0”, the TM can immediately tell the input is not acceptable, by terminating 
there. The TM will also need to record whether there will be carry. This can be done by using a 
distinct set of states. Thus, the TM distinguishes the following four acceptable cases: “0 + 0 = 
0”, “0+ 1=1",“1+0=1", “1+1=0" (with carry). Since the only information the TM needs 
for the next round is the presence/absence of carry, the processed bits can be replaced. Using the 
following replacement pattern, the tape configuration becomes analogous to the original 
configuration (except that there are repetitive delimiters, ‘+’ and ‘=’). 


.. OD§ Aggie... 


Then, we have an analogous, but smaller problem (cf. divide and conquer). The current focus “1 
+ 1 =0” is correct if we process the carry bit in the next around. This TM contains two similar 
submachines: one for the case with no carry and the other for the one with carry. As done in 
Eric’s machine, it is possible to encode the distinction on the tape (with fewer states, but more 
complex tape operation). Since the argument bits are already consumed, the TM interprets 
missing bit as 0. Then, “carry +0+0=1”. 


.. OODR+AI-2goo... 


If the input survived this much, all the bits are consumed. The TM can then check for this by 
scanning only ‘+’ and ‘=’. 


..-ODD+++===8 gp... 


The transition diagram is on the next page, and the code (nk-adder.jff) is available in our JELAP 
folder (http://www.tcnj.edu/~komagata/csc460/05s/JELAP/). One unusual aspect of this version 
is that the input “+=0” will also be accepted because missing arguments across ‘+’ are 
interpreted as 0. I tested with various inputs, but I have no doubt you can come with problematic 
inputs. If you test-drive this TM and identify a bug (or be convinced that it is correct), you will 
receive some reward. 
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// End 
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Unit A4 Supplement, 2/1/05 


Here are sample response to Exercise A3 Part 1 


Task 1 (Program Verification): One of the software industry’s biggest concern is how to check 
whether their products are really correct with respect to the specification that the developer and 
the user both agreed. According to a variety of sources, inability to do so will cost us an 
incredible amount of money in the future (well, this must be already happening). As you know, 
generality is a prime concern in Computer Science. So, why don’t we write a single program 
that could verify whether the program correctly solves a problem with respect to a given 
specification? Give the set representation of the computational problem involved here. 
Speculate the basic Theory properties (i.e., possibility to solve computationally, the simplest 
mechanism, practicality with large data), referring to the set (a concise, informal description 
suffices). Can you think of such a program? 


{(s, p) | Specification s specifies program p} 

{(s, p, i) | Specification s specifies program p when run on input i} 

Note 1: It would be difficult to make the notion of “specification” more precise for this exercise. 
Note 2: Including collections of specifications and programs in this representation would view 
the problem at a level higher than the original. 


Task 2 (Map Coloring): It has been shown that with four distinct colors, we can color any map 
so that the neighboring countries (or whatever political boundaries) do not share the same color. 
With three colors, we may or may not be able to do the same thing. Give the set representation 
of the computational problem involved here. Speculate the basic Theory properties, referring to 
the set. 


{(R, C, N, a) | 
R is a non-empty set of regions, 
C is a non-empty set of colors where |C| = k (for some positive integer k), 
N is a binary relation on R specifying the neighboring regions, 
a is a function from R to C assigning a color to a region, 
To specify different colors for two neighboring country: if (x, y) € N, a(x) #a(y)} 


Fog, (UP ce Pols (Cina Cale (1 Pal A Pees) UU C2), a Od) we) ed 


// End 
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Exercise A4, 2/1/05 


Part 1: Test-Drive Turing Machines 


Once your problem is represented as a set (precisely and discretely), you can analyze various 
computational properties in a formal manner. The standard tool to analyze “computability” (e.g., 
whether an algorithm exists for the problem) is the Turing machine (TM). By understanding this 
tool, we can access the rich literature and also make a lot of connections with other forms of 
analytical tools. So, in this exercise, you test-drive them, using a simulator called JELAP 
(http://www.cs.duke.edu/~rodger/tools/jflap/; links are clickable on-line; also linked from the on- 
line syllabus). A copy of the program (JFLAP. jar) and some examples (*.TM, *.TM.jf£) are 
available locally (http://www.tcnj.edu/~komagata/csc460/05s/JELAP/). 


Task 1: Using JFLAP, design a TM that would 
loop regardless of the input. Choose one of the 11K f\onr . (ELE om 
following options to show your work: (1) _ Ah 
attach/copy/draw (the screen shot of) the - 
transition diagram of your TM, or (2) include 

the formal definition of your TM. In addition, 
concisely explain how your TM would work. 


T™ M=(Q,2%,T, 6, go, O, F) 

Q: finite set of states 

2: finite set of input symbols, 6 ¢ X 

T: finite set of tape symbols, be T, 2cT 

5: transition function Q x > Q xT x {Left, Right} 
qo: the start state qo € Q 

4: blank symbol (B in the text) 

F: set of accept states F c 0 


Task 2: Consider a problem represented as 
{x+y=z | when x, y, and z are interpreted as non- 
negative binary numbers in some standard 
representation, the arithmetic relation x + y = z 
holds}. In this representation, you may consider 
“xty=z” as a variant form of (x, y, z) with special delimiters. Repeat the process as in Task 1. 
Note that this task will be challenging, tedious, or both. You can stop when you feel you spent 
enough time. 


Task 3: Define an interesting, non-trivial problem of your own as a set. Then, repeat the process 
as in the earlier Tasks. Again, you can stop when you feel you spent enough time. 


Part 2 (optional): TM Variations 


Note: Depending on your time and interest, you decide whether to do this part. 


If you look up references, you will occasionally find different versions of TMs. Initially, it 
might be confusing. However, if you think carefully, it is possible to see those variations as 
different ways of representing the same notion of “computation.” But what does it mean to be 
the “same?” We will discuss this point more in detail in Module B. This part of the exercise is a 
preview and can be considered as an extended exercise on TMs. 


Task 1: In another Theory of Computation textbook (Sipser, 1997), Turing machines are defined 


differently from our version on the following points: 
1. Use of a semi-infinite tape. The input string must be placed at the left end of the tape. 
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2. There are exactly one accept state and exactly one reject state. 
Despite the difference, we can still say that this version is “equivalent” to our version. Try to 
explain how to establish such equivalence. 


Task 2: Most “decent” extensions of the standard TM (cf. Text Sec. 8.4) are known to be still 
equivalent to the standard version (i.e., our version). Can you think of an extension of the 
standard TM that could do more than what the standard TM can do [cf. my response to your Unit 
A3 Summary Questions Q6/A6]? Even if you cannot come up with such an extension, can you 
still describe how to analyze whether a certain given TM can do more than the standard TM? 


Survey: Time spent between classes: 


// End 
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Questions/Review 


° Ex A4 
— Part 1: TM 
* Binary addition 
+ Your own 
—Part 2 
+ Equivalence with an alternative version 
+ Beyond TM 
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Problems du Jour 


* Vending machine 

* Compilation 

* Termination detection 
(“halting problem’) 

¢ Infinite-loop detection fecniinete? 

¢ Weather forecast 2. Impossible to 

« Jackpot prediction solve? 

* Driving 
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Properties 
1. Always 


Overview: Theory of Computation 


impossibility Computability 


Uninteractable? 


Focus in Modules A & B 


Interactable? 
ee 
Distributivity? 


real practicality 


intractable Context-free 
iy 


Complexit Tractable Regular 


Languages 


~ 
practicality 
Automata 


simplicity 
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Unit A5: Overview 


Discuss problems of the day 


Introduce “Computability” properties used 
to classify problems with respect to TM 


behavior inuitn tt more precise csoussion in Module 8 


Discuss the role of proof in this course 
Preview Module A Evaluation Workshop 
Preview Exercise A5 Module A 
Comprehensive Exercise 
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Problems and Properties 


Single problem 


€.g., (xX, y, Z) 
Input (tuples): ° e¢ ¢ Input (problems): © ° e 
| I 


Classifying problems 


/ 


yes 


Problem: {( ) | ...} Property, e.g., computability 
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Decidable Problems 


* Property 
— Some TM always terminates and either 
accepts or dies (rejects). 
« Examples 
— Binary number addition 
— Palindrome detection 
— Vending machine 
— Compilation 
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Decidability Synonyms 


Decidable [of a procedure, problem] 


TM/algorithm exists, i.e., always 
terminates 


TM-decidable [of a TM] 

Computable [of a procedure/function] 
Solvable [of a problem] 

Recursive [of aset], cf. recursive function 
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Undecidability Synonyms 
« Undecidable 


¢ Unsolvable 


* Note: Incorrect definition of “unsolvable” 
on the earlier version of “Topics” doc. 
Please fix. | apologize. 
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TM-Recognizability Synonyms 


¢ TM-recognizable 
¢ TM exists (but not necessarily always terminates) 
e Recursively enumerable (RE) [ofa set] 
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Undecidable Problems 


Property 


— Not decidable (complement of decidable 
problems) 


— Equivalently: Not always terminate, or not 
always end up in either accept state or die 


Examples 
— Halting problem 
— Floor tiling 


Decidable 
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TM-Recognizable Problems 


¢ Property 
— Some TM can identify all the acceptable 
inputs 
— Note: OK if loops on non-acceptable inputs 
* Example 
—Allofdecidable  Ungecidable 
problems 
— Halting problem 
(undecidable) 
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Non-TM-Recognizable Problems 


* Property 

—No TM can represent 
¢ Example 

— Infinite loop detection 


— Weather 
Undecidable 
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Non-TM-Recognizability Synonym 


* Non-TM-Recognizable (non-TM-rec.) 
¢ Non-RE 
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Property of “Co-” 


* co-X 
— Consider the complement of a problem 
— If the complement has property X, the original 
problem is called co-X. 
« Examples 
— Infinite-loop detection is co-TM-recognizable. 


l.e., the complement problem (halting 
problem) is TM-recognizable. 


— Palindrome detection is co-decidable. 
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Review: Proofs 


¢ Proof [Text 1.2-1.4] 


—A sequence of justifiable steps of deriving a 
theorem from axioms and other theorems 
using rules of inference 


¢ Example proof technique 
— To prove a statement: X = Y 


—(1) Prove X = Y, and (2) Prove X = Y 
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Semi-decidable Problems 


Equivalent terms 
* Semi-decidable 
* TM-recognizable but not decidable 


Non-TM-rec. 


\ TM-recognizable 
« | Decidable 


Undecidable 
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Useful Theorems 


* Theorem: Decidable <= TM-recognizable 
and co-TM-recognizable 
— Note: Theorem as a statement in a theory, 
which needs to be proved 
¢ Corollary (a theorem easily derivable from 
another theorem): The complement of a 
decidable problem is decidable. 
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Unit Summary 


¢ With respect to impossibility, problems can 
be classified into 3 disjoint sets, based on 
whether a TM exists and/or always gives 


an answer. + Vending machine 
+ Compilation 
+ Halting problem 
+ Infinite-loop detection 
; Non-TM-rec. + Weather forecast 
Undecidable . + Jackpot prediction 
+ Drivin 
_/Bemi-decidable 9 
TM-recognizable 
Decidable 
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Preview 


Module A Evaluation Workshop 
* Required materials (hardcopy) |printin advance 


— Module A evaluation form 


— Supporting notes "Significance of supp. notes? 
— Exercises 
* Activities 
— Module review exercise 
— Peer discussion 
— Reflection and self-evaluation 
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Summary Question 


¢ Did you get the big picture of the 
“computability” part of the Theory of 
Computation? Explain. 

* Questions/Comments/Suggestions 


(make sure to understand the 
materials and also the evaluation 
procedure) 
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Exercise A5 (Module A Comprehensive Exercise), 2/4/05 


Part 1: Mayan Script (Sample Problem 2) 


In this exercise, you will practice many aspects involved in this module, using a quasi-practical 
problem of deciphering Mayan scripts. As you do this exercise, examine your understanding 
with respect to the evaluation criteria on the form. As noted on the form, in order to demonstrate 
your understanding, you may refer to your exercises (including this one), in your supporting 
notes. Note that providing a correct answer is not the focus of this (and other) exercise(s); in 
many cases, there won’t even be a correct answer. Do you best to apply what you learned (as if 
you are developing a theory from axioms and rules of inference at a meta-level). 


Problem 

You found an ancient Mayan script that is supposed to indicate the location of hidden treasures. 
The only resources in this ancient language you have access to are: (1) an extensive list of 
synonyms and (2) a collection of sentences whose meaning are already known. 


For example, let’s imagine that you need to translate the following sentence (obviously, not real 
Mayan): “koregawakarukane” with the following resources: 


Synonym list: “korega” = “maa”, “ruka” = “nnee’, “neene” = “darona’, etc. 
99 66 


Sentences whose meaning are known: “maawakanneedarona’”, “mosikasitarawakarukamone’’, 
etc. 


A sample session of analysis/translation by substituting synonyms would be (underlined words 
are replaced with italic words): 
“koregawakarukane” — “maawakarukane” — “maawakanneene” — “maawakanneedarona” 


Then, we can tell that “koregawakarukane” means the same thing as “maawakanneedarona.” 
The latter sentence is among the known, so we should be able to understand the former as well. 
If there are many sentences to analyze and the synonym list is extensive, we will naturally think 
of doing the task with a computer. Unfortunately, it is known that there is no algorithm for this 
problem. 


Hint: Try to use schematic diagrams! 

Task 1: The collection of sentences whose meaning can be deciphered (in this problem) may be 
defined as a “theory” derived from axioms and rules of inference. Identify the axiom(s) and 
rule(s) of inference. 

Task 2: Represent this problem as a set. Use the predicate notation and try to make the 


description part (the right of ‘|’) as precise as possible so that each instance could be used as an 
input to some Turing machine. 
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Task 3: It has been known that this problem does not admit any algorithm. This suggests that 
any general programming attempt must be either wrong or forced to loop on at least some input. 
Suppose that you came up with a Turing machine that makes no errors but could loop. 
Informally describe (part of) the mechanism/behavior of the TM. 


Task 4: Since there is no algorithm, this problem is not decidable. But what about the other 
properties/classes: undecidable, TM-recognizable, non-TM-recognizable, and semi-decidable? 
For each of these classes, analyze/explain whether this problem belong to it (1.e., say whether or 
not the problem is undecidable, TM-recognizable, etc.). 


Part 2: Evaluation Form and Supporting Notes 


Review the evaluation procedure. Then, complete your evaluation form and supporting notes. 
Your supporting notes are supposed to reflect and communicate your thoughts. This type of skill 
is essential for every one of us to be able to improve from what we are now. Bring them to the 
evaluation workshop (hard copy). Print them well in advance so that you can avoid potential 
problems, e.g., not being able to print just before the evaluation. 


Survey: Time spent between classes: 


// End 
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Module A Review Exercise, 2/8/05 


Do one of the following problems. Write your response on back or on another sheet of paper. 
Three computability properties to be used: decidable, semi-decidable, non-TM-recognizable 


Option 1: Floor Tiling (Sample Problem 3) 


Recall the “floor tiling problem.” That is, you will be using n types of square tiles with distinct 
patterns to fill the space assigned to you, which could be in any shape. You need to match the 
edges of the tiles as shown in the left example below (in this case, using three types of tiles). If 
you arrange your tiles as in the right example below, you will be considered “‘esthetically- 
challenged.” [Images from Algorithmics by David Harel] 


(1) 


Dill <b ap, "4 
A atl Nal 


(1) (2) (3) 


the 


Ing. 
d a> i 


ane 


Dsl ”SPdd, "4 
Nel A al 


In class, we briefly noted that it would not be possible to write a program to tell whether or not a 
given set of n types of square tiles can fill your floor with the above-mentioned condition (note: 
take this information as given). Classify this problem with respect to the three computability 
properties listed above. Explain. 


Option 2: Arithmetic System (Sample Problem 7) 


One of the great achievements in logic is that it can formalize a wide variety of systems. For 
example, the mathematical system of arithmetic can be formalized in first-order logic in a 
consistent (roughly, meaningful) way. Its basic components include axioms such as “for any x, x 
+ 0=x” and “for any x, x x 0 =0”. In this system, we can write all sorts of formulas that are 
representative of arithmetic. One property that a good logic system must have is that all “true” 
formulas can be proven from axioms using rules of inference. But it has been shown that not all 
true formulas in such an arithmetic system can be proven (note: take this information as given). 
Classify this problem with respect to the three computability properties listed above. Explain. 


// End 
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Exercise A6/B0, 2/8/05 


Part 1: Simulating a TM Using a TM 


In Module A, we discussed our intuition that the halting problem is semi-decidable and infinite-loop detection is 
non-TM-recognizable. In either case, it would be impossible to create an algorithm. So, we would not attempt to 
write a program to solve these problems in the general case. However, by no means we “proved” our intuition. Is 
our intuition really correct? In Module B, we will explore a more convincing argument behind the intuition, so that 
we can discuss other (possibly similar or dissimilar) problems with more confidence. 


In this exercise, you need to deal with a certain type of “circularity.” As a computer science student, you have been 
exposed to the idea of “recursion.” So, you must know that you need to face circularity in a principled manner. As 
a related example, you might consider the case of “bootstrapping” in compiler construction. Probably the most well- 
known example is the fact that the C language is “written” in C. Have you heard of it? Did you think about this 
carefully? If you are interested in, see the Appendix at the end of this exercise. Note: One of the C language 
developers, Brian Kernighan, will be giving a keynote speech on Sat., Apr. 16, 2005 during Trenton Computer 
Festival (http://www.tcf-nj.org/), here at TCNJ. 


Hint: Try to use schematic diagrams! 


Task 1: Suppose that you are given a program P written in, say, Java which is claimed to do the following (i.e., for 
this task, you must accept that this claim holds): given another program M in Java (e.g., the source code in ASCII) 
along with some input J to M, P can determine whether M terminates on /. First, can you think of how P might 
work? Since M can be any program, we can supply (the source code of) P itself as the input to P (i.e., M = P). 
What can you conclude? 


Task 2: To discuss problems such as the halting problem using TMs, we need a way to represent a TM as a string so 
that it can be placed on the tape of a TM. Then, we can write the representation of a TM M along with some input / 
to M (e.g., separated by ‘#’) on the tape of some TM P. First, propose a way to represent a TM as a string. As in 
Task 1, we can supply P itself and J to P. Since we have intuition that this problem is semi-decidable, we would 
expect that P may never terminate on some inputs. Try to compare this situation with your analysis in Task 1, 
applying the materials discussed so far in this course. 


Part 2: Mini Research Project Ideas 


At the beginning of the semester, you were asked to come up with and continue to entertain your own practical 
problems through the course. While your problems were good starting points, our experience in this course could 
still be enriched, if we discuss even more examples. So, we will explore some additional problems as mini research 
projects. This time, you will be asked to identify one or more research questions (from the list below) and write a 
concise essay about them. Later, you will be asked to do some research activities related to your problems. Note 
that some of the sample questions are not precisely stated, some of them are probably not solvable, and some of 
them more or less solved. The instructor will be able to help you find relevant reading for some problems, but not 
all. 


Task: Identify one or more research questions (or some variants of those) from the list below. Write a concise, 
possibly speculative essay about the question(s), e.g., why you are interested in, what your guessed answer is. 


List of sample research questions 

Can organizational dynamics be modeled as an algorithm? 
Can evolution be modeled as an algorithm? 

Can ecology be modeled as an algorithm? 

Can human development be modeled as a computer? 

Can our minds be modeled as a computer? 

Can vision be modeled as an algorithm? 
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Can learning be modeled as an algorithm? 
What would be the minimal mechanism to process human language? 
Can the entire situation of an arbitrary game be modeled as an algorithm? 


. Would “perfect” user modeling, e.g., for web search, be possible? 

. Would “perfect” computer security be possible? 

. Can the entire process of software engineering be modeled computationally? 

. Can computer networks be modeled as a single computer? 

. Could biology be reduced to physics? 

. Could some computer generate real (not pseudo) random numbers? 

. Would it be possible to decide whether the given numbers are random? 

. Would randomization affect computability and/or complexity? 

. Would parallelism affect computability and/or complexity? 

. Would artificial neural network be more powerful than TMs? 

. Would cellular automata be more powerful than TMs? 

. Would the use of analog (or fuzzy) values affect computability? 

. Would relativistic, quantum, or some other modern-physics-based computation surpass TMs? 
. Can all the cases of on-line algorithms be simulated by off-line computation? [On-line algorithms would 


obtain inputs as the time progress. Off-line computation would provide all the possibilities as input at once. Cf. 
function-to-relation conversion used to fold the output within the input] 


. Would oracle computing affect computability? [Oracle computing: A TM with the capability to ask questions 


to another mechanism] 


. Would persistent TM be able to compute more than the standard TM? [Persistent TM: Multiple sessions of 


TM operation with some memory between them] 


. Would accelerating a TM give more power? 

. Would slight error tolerance affect any aspect of the Theory of Computation? 
. What would be the ability of a finite automaton with a queue? 

. What would be the effect of “constant” (as in complexity analysis) in practice? 
. Can any mathematical function be represented computationally? 

. What exactly are power sets doing to the Theory of Computation? 

. Is what you can do in logic the same as what you can do with computation? 

. If you have a research question of your own, please consult the instructor first. 


Appendix: Bootstrapping in Compiler Construction 


Languages involved in compiler construction Source language -— Compiler —J Target language 


Language used to 
write the compiler 


Write a compiler for the minimal C in SPARC assembly minimal C }— Compiler —4 SPARC Code 
(cf. the “base” case in recursion) 

SPARC Assembly 
Write a compiler for fud/ C in the minimal C full C -— Compiler —4 SPARC code 


(still generates SPARC code) | ee 
minimal C ;-— Compiler —+ SPARC Code 


SPARC Assembly 


Special question (completely optional): Could we “bootstrap” to develop a programming language that is 
“verifiable” (cf. Ex A3 Part 1)? Those who respond to this question with an interesting idea (on the discussion 
board) will receive a free book (areas: discrete math, logic, theory, algorithms, etc.). 


Survey: Time spent between classes: 


// End 
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Module A Evaluation 


Review your portfolio 


* My comments are sporadic and scattered 
on Review Ex, Comprehensive Ex, and 
Supp. Notes. 


You are encouraged to clarify and discuss 
my comments. 


You can keep the folder till the next class; 
then, return it to me. 
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Overview: Theory of Computation 


impossibility Computability 


Uninteractable? 


Focus in Modules A & B 


Interactable? 
Za 
Distributivity? | Undecidable 
as 


Decidable 
z= — 


real practicality : 
Intractable  Context-free 


Complexity Tractable Regular 


~ 
practicality Languages simplicity 


Automata 
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Module B Content Goals 


CG4 Computability 

+ Universal TM 

+ Diagonalization technique 

+ Halting problem and infinite-loop detection 

+ “Reduction” 

* Church-Turing thesis 

* Can explain (i.e., teach) this goal to CS students outside this class. 

CG7 Power set 

+ Understood (i) that the power set is always “larger” than the original 
set and (ii) why the power set of a countable set is uncountable. 

CG3 Interactive computation 

+ Limitations of the traditional, algorithmic approach to “computability.” 
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Unit B1: Overview 


Understand Module B learning goals 
Review Module A 

— Comprehensive/Review Exercises 

— Practice 


Understand a systematic way of simulating 
a TM using a TM 


Preview Exercise B1 “Universal TM” 
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Module B Overview 


B1 Computability: Universal TM 

B2 Diagonalization: Math techniques 

B3 Halting Problem: The main problem 

B4 Reduction: Comparing problems 

B5 Church-Turing Thesis: Equivalence 

B6 Evaluation Workshop (similar to Mod A) 
B7 Review (TBA) 
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Module B Performance Goals 


PG4 Critical attitude 


* Critically analyzed the following points: 


i) the infinite tape in TM 

ii) the diagonalization technique 
iii) no tolerance to errors 

iv) the lack of interaction in TM 
(v) any other aspects 


( 
( 
( 
( 


* Critically analyzed the usefulness of the “computability” 
* Critically analyzed the course materials, cf. your learning 


PG6 Initiative 


* Chose a research question for the mini research project and started 


to explore it. 
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Module A Comprehensive Exercise 


Unknown: “koregawakarukane” 


— Synonym list: “korega” = “maa”, “ruka” = “nnee”, 


“neene” = “darona”, etc. 


— Sentences whose meaning are known: 
“maawakanneedarona’”, 
“mosikasitarawakarukamone’, etc. 


Analysis/translation: “koregawakarukane” > 
“maawakarukane” > “maawakanneene” —> 
“maawakanneedarona” 
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Module A Review Exercise 


Decidable, Semi-decidable, or Non-TM-rec.? 
* Option 1: Floor Tiling 
* Option 2: Arithmetic System 
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Review 


Computability Classes 


With respect to impossibility, problems can 
be classified into 3 disjoint sets, based on 
whether a TM exists and/or always gives 
an answer. 
Undecidable pee 


Semi-decidable 
TM-recognizable 
Decidable 
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Module A Comprehensive Exercise 


+ Task 1: The collection of sentences whose meaning can be deciphered (in 
this problem) may be defined as a “theory” derived from axioms and rules of 
inference. Identify the axiom(s) and rule(s) of inference. 

+ Task 2: Represent this problem as a set. Use the predicate notation and try 
to make the description part (the right of ‘|") as precise as possible so that 
each instance could be used as an input to some Turing machine. 

+ Task 3: It has been known that this problem does not admit any algorithm. 
This suggests that any general programming attempt must be either wrong or 
forced to loop on at least some input. Suppose that you came up with a 
Turing machine that makes no errors but could loop. Informally describe 
(part of) the mechanism/behavior of the TM. 

+ Task 4: Since there is no algorithm, this problem is not decidable. But what 
about the other properties/classes: undecidable, TM-recognizable, non-TM- 
recognizable, and semi-decidable? For each of these classes, 
analyze/explain whether this problem belong to it (i.e., say whether or not 
the problem is undecidable, TM-recognizable, etc.). 


CSC460 B1 


Group Exercise 1 (Review) 


Consider a couple whose characters are 
completely opposite. The woman notices the 
positive aspect of everything they encounter, 
while the man notices the negative aspect. Note 
that there is no intermediate state. 
1. Characterize the woman and the man with respect to 
the computability classes 


2. Give black box representations of TMs M, and M, 
that captures the behaviors of woman and man, resp. 


3. Represent the behavior of the couple as a whole by 
properly connecting M@, and M, 


eae ‘schematically... , 


Review 
Problems and Properties 


Single problem 


e.g., (xX, y, Z) 
Input (tuples): ° e @ Input (problems): © ° e 


n 


Classifying problems 


| 


Problem: {( ) | ...} Property, e.g., computability 
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Property of “Co-” 


co-X 
— Consider the complement of a problem 


— If the complement has property X, the original 
problem is called co-X. 


Examples 
— Infinite-loop detection is co-TM-recognizable. 


l.e., the complement problem (halting 
problem) is TM-recognizable. 


— Palindrome detection is co-decidable. 
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Complementary TMs 


Decidable <> TM-recognizable and co-TM-recognizable 


Moos accept |———> positive 


oN 


Lr 


Input (tuples): 


Mneg accept |—-* negative 
_ 


TM-rec 


decidable 
osc460 B1 co-TM-rec 


Object vs. Meta-Level 


Compiled language vs. compiler language 
Problem vs. properties 

—E.g., computability 

Formal logic vs. informal reasoning 

Your thinking vs. your thought about your 
thinking 

—E.g., course activities vs. supporting notes 
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Review 
Useful Theorems 


Theorem: Decidable <= TM-recognizable 
and co-TM-recognizable 


— Note: Theorem as a statement in a theory, 
which needs to be proved 


Corollary (a theorem easily derivable from 
another theorem): The complement of a 
decidable problem is decidable. 
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Ex A6/BO 


¢ Part 1: Simulating a TM Using a TM 


— Task 1: Program simulation using a computer 
— Task 2: TM simulation 


¢ Part 2: Mini Research Project 


— Topic(s)? 
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Circularity 


Recursion 
Self-reference 
Reflection 
Self-evaluation 
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“This sentence is false.” 


¢ The truth value of the above sentence? 
¢ What makes it paradoxical? 
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Universal TM (UTM) 


* Given (i) an encoding of the formal 
definition of a TM, M, and (ii) an input 
string, i, to M on its tape, simulates the 
behavior of M on i. 


—l.e.. specification of a TM through the formal 
definition on the tape. 


* Accepts if M accepts i. 
—|.e., exactly mirrors the behavior of M. 
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The Main Problems, Revisited 


Halting problem: {(, i) | M halts on i} 

— Using a UTM, simulate M’s behavior on i 
— If M halts on i: Accept 

— Otherwise: How can we tell? 

Inf.-loop detection: {(M, i) | M loops on i} 
— Using a UTM, simulate M’s behavior on i 

— If M loops on i: How can we tell? 

— Otherwise: dies 
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Group Exercise 2 


* Come up with a statement involving TM(s) 
corresponding to the liar paradox in the 
following sense: ‘This sentence is false.” 

—A TM would either halts or loops. 
— The statement is contradictory regardless of 
the TM’s halting/looping status. 


— The statement shows the undecidability of 
one or both of the problems. 
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UTM Specifics 


Tape input 

— ...4TM formal defftinput to the TML... 

Mechanism 

— Allocate an unused area of the tape to represent the 


working tape for the TM (if more space is required, 
move the surrounding areas outward) 


— Allocate an unused area of the tape to keep record of 
the internal state of the TM 


— Interpret the TM definition and simulate its behavior 
on the input to the TM 
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Unit Summary 


Module B learning goals 
Module A review 


Liar paradox and undecidability 
Universal TM (UTM) 
Preview Exercise B1 “Universal TM” 
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Midterm Survey 


¢ To be administered and collected by 
a volunteer 
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Exercise B1, 2/11/05 


Part 1: Universal TM (UTM) 


In order to convince ourselves that halting problem and infinite-loop detection are undecidable, 
we need to use the notion of UTM. The following tasks may appear complicated due to the 
involved nature of “universality.” If you have questions, please send them to the instructor. If 
you still have difficulty for any of the following tasks, explain your situation. 


Task 1: The essence of UTM is its ability to deal with any TM, including itself. Thus, it 
becomes possible to discuss circularity. Come up with and explain examples circularity not 
discussed in class (no need to be in computer science). 


Task 2: Recall how a UTM can be organized (see the slide on UTM Specifics). However, this 

may not be the only way. Analyze whether the following two options are viable as a UTM 

mechanism: 

A. Instead of “interpreting” on-line, “compile” the given TM specification off-line and run the 
compiled machine. 

B. Identify all the possible TM’s with a unique natural number. Also define the UTM so that 
given the ID of a TM, it can simulate the behavior of the TM. Then, instead of the formal 
definition of a TM, provide just the ID (still along with its input). 


Task 3: Read the attached chapter on “The Halting Problem” from “The Turing Omnibus” (see 
our reference list for more info about this fascinating book, available in the library). Explain the 
role of UTM in this chapter. In addition, describe whether you were convinced of the claim that 
the halting problem is undecidable. You must be very critical about the arguments in the chapter 
and your own thinking. 


Task 4: Write up your (possibly revised) response to Group Exercise 2 (Unit B1), i.e., a 
statement involving TM(s) corresponding to a version of the liar paradox, e.g., “This sentence is 
false.” This time, though, you can also refer to the reading in Task 3. In addition, you must 
address the following point. In Task 3, the use of the UTM was crucial. If there is some 
connection between the liar paradox and the halting problem, we must be able to see the 
analogous concept in the liar paradox. Then, is the notion of “universality” involved in the liar 
paradox? 


Part 2: Review “Module A”’ 


Review the materials from Module A, including Comprehensive, Review Exercises, and Group 
Exercise 1 (Unit B1). If you were entirely sure about these, skip this part. If you were able to 
improve your understanding through the review process during class and/or in this part of the 
exercise, concisely note the improvement. 


Survey: Time spent between classes: 


// End 
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Ex B1 


* Part 1 UTM Review question: describe UTM 
— Task 1: Circularity examples 
— Task 2: A: Compiling a TM, B: ID’ing TMs 
— Task 3: UTM in the reading 


— Task 4: TM version of the liar paradox; — l.e.. specification of a TM through the formal 
universality definition on the tape. 
Part 2 Review 


* Accepts if M accepts i. 
—|.e., exactly mirrors the behavior of M. 
Ex BO Part 1 Task 1 phrasing 


Universal TM (UTM) 


* Given (i) an encoding of the formal 
definition of a TM, M, and (ii) an input 


string, i, to M on its tape, simulates the 
behavior of M on i. 


CSC460 B2 CSC460 B2 


UTM Specifics 


UTM, Schematically 


Tape input 


Machine with a finite number of states 
— ...TM formal defttinput to the TM... 
Mechanism 


Some mechanism to control 
State 0 | State 1 | State 2 State k 
— Allocate an unused area of the tape to represent the 
working tape for the TM (if more space is required, 
move the surrounding areas outward) ..({q0,q1},{0,1},{E1},{((q0,0),(q1,0,R)),((q0,1),(q1,1,R))},q0,E1,{q1})#01000... 
— Allocate an unused area of the tape to keep record of formal definition of the input-detection M_ “input to M_ 
the internal state of the TM ; : ; 
— Interpret the TM definition and simulate its behavior The right of the input to be used to simulate 
on the input to the TM + Working tape for M 
+ Internal state of M 


CSC460 B2 CSC460 B2 


Unit B2: Overview 


¢ Understand mathematical tools for ¢ Cf. Ex B1 Part 1 Task 2B 
understanding undecidability 


* Possible to assign a unique ID (natural 
— Explore ID’ing TMs with natural numbers number) to every TM? 
— Explore the notion of “counting” 


— Introduce the “diagonalization” technique 
¢ Understand the “diagonalization language” 
* Preview Exercise B2 “Diagonalization” 


Labeling TMs 


CSC460 B2 


CSC460 B2 


Thinking Big ... 


How many TMs are there in the world? 

How many inputs must a TM deal with? 

— How many different possibilities must a TM 
deal with? 

Mismatch between the collection of TMs 

and the collection of input possibilities = 

Undecidability? 

Mathematical support for analyzing 

computability 


CSC460 B2 


Group Exercise 1 


« Analyze whether the following sets are 
countable 
. Set of integers 
. Set of rational numbers 
. Set of real numbers 
. Set of all TMs 
. Set of all strings 
. Set of all languages (set of sets of strings) 


7. Power set of some given set 
CSC460 B2 


Uncountable Sets 


Impossible to order along with natural numbers 
* Set of real numbers 

— Diagonalization [Cantor] 
¢ Power set of natural numbers 

— Same “density” as real numbers 


* Set of all /Janguages (i.e., set of sets of 
strings, or power set of all strings 


CSC460 B2 


Set Cardinality 


Informally, the number of elements in a set 
* Countable 

— Finite: e.g., |{yes, no}| = 2, || =0 

— Countably infinite: e.g., N (natural numbers) 
¢ Uncountable: If not countable 


CSC460 B2 


Countable Sets 


Possible to order along with natural numbers 
* Any finite set 

Set of natural numbers 

Set of integers 


Set of rational numbers (represented as 
fractions) 


Set of all strings 
* Set of all TMs 


CSC460 B2 


Cantor’s Diagonalization 


¢ Suppose that all the real numbers can be 
ordered linearly. » Digits below decimal 
! 4 


j 


' 

. , Only 0 and 1 are used 
(possible to use any 
numeral 0 through 9) 


List of real numbers | | , 
, 


taportal 


¢ There still is a number not covered. 
* The assumption is wrong. Impossible. 


cSC460 B2 12 


Proof by Contradiction Proof by Contradiction, Formally 


To prove: Statement X Hypothesis: m <> 0, =m > i, =i 

« Assume the negation of X, i.e., not X Proof of m from the above hypothesis 
* Derive a contradiction 1. _ [hypl 
* Then, by this proof technique (proof by : [hyp to be rejected] 
contradiction) X must be true (because X - (|b [Modus Ponens: 1, 2] 
is either true or false) : i [hyp] 
[contradiction: 3, 5] 
[proof by contradiction: 2-5] 


CSC460 B2 CSC460 B2 14 


Power Set (of set A): P(A) Interim Summary 


¢ The set of all the subsets of A, i.e., P(A) = Power set always increases the cardinality. 
{X|X CA} The power set of a countable set: uncountable 

* Examples ee The set of TMS: saree as) 
— P({0, 1}) = {@, {0}, {1}, (0, 1}} The set of strings: countable 
— P({0}) = {@, {0}} The set of languages (specifies which string to 
-/°@)=9 be included): power set of the set of strings > 


i uncountable 
; ae ie) a There won't be enough TMs to cover all 
— |P({O, 1})| =|{0, 1}? = 27 =4 


languages = The source of undecidability solution? 
— |P(N)| =? ct. pigeon hole principle: 
CSC460 B2 Power set in the real world? 15 CSC460 B2 The halting problem explained? 16 


Agenda Group Exercise 2 


¢ Rest of today: Identify an example * Identify an example language which no 
language which no TM can decide (there TM can decide (there must be uncountably 
must be uncountably many of those...) many...) 

¢ Next class: Using this language, prove the ¢ Hint: Consider the collection of all 
computability classes of the main problem languages; Recall Cantor’s diagonalization 
(e.g., halting problem). 


CSC460 B2 CSC460 B2 


Diagonalization Language (L,) 


» List of strings 
4 


estes 2a 
Listof TMs |.) 
’ 


* Entry 1: M; accepts w, 
* Entry 0: M; does not accept w, 
* L,= {w;|M; does not accept w;} \No TM accepts! 


CSC460 B2 19 


Main Problems 


The halting problem is semi-decidable. 

HALT yy = {(M, w)| TMM halts on w} ww for word" (input) 
Infinite loop detection is unsolvable. 

LOOP yy = {(M, w) | TM M loops on w} = (HALT yy)’ 
Universal language is semi-decidable. 

ACCEPT yy = {(M, w)| TM M accepts w} 

The complement of universal language is 
unsolvable. 

NACCEPTyy = {(M, w)| TM M does not accept w} = 


(ACCEPT 1,)' 


CSC460 B2 21 


Midterm Survey 


¢ To be administered and collected by 
a volunteer 


CSC460 B2 


Diagonalization Language 


* The diagonalization language is non-TM- 
recognizable because no TM would accept 
it. 


CSC460 B2 


Unit Summary 


Set cardinality: countable vs. uncountable 
Power set: increases the cardinality 


Diagonalization: shows uncountability of a 
set (using proof by contraction) 


The source of undecidability: 
uncountability 


The diagonalization language, L, 


CSC460 B2 


CSC460 


Name: 


Exercise B2, 2/15/05 


Part 1: Diagonalization 


Today, we discussed that for countably-many TMs, there are uncountably-many languages. 
Thus, no matter how hard we try, there won’t be enough TMs to cover all of the languages. This 
immediately suggests that the acceptability problem, i.e., {(M, i) | M accepts i}, is undecidable. 
Among the many missing points in the uncountable set, we also identified a non-TM- 
recognizable set (problem), the “diagonalization language.” Regardless of the nature of a 
problem, once represented as a set, the notion of countability offers an insight into analyzing 
computability classes. In this respect, understanding diagonalization is important. We will 
practice this technique along with proof by contradiction with a few more examples. 


Task 1: Power set of natural numbers: Show that the power set of the set of natural numbers is 
uncountable, using diagonalization. 


Task 2: Languages: We discussed that the set of all languages is uncountable relying on the fact 
that power set increases the set cardinality (i.e., the power set of a countable set is uncountable). 
For this task, show (the same result) that the set of all languages is uncountable, directly using 
diagonalization. 


Part 2: Review “Discrete Math’’ 


If necessary, review the following concepts in Discrete Math: 


e Set cardinality 

@ Power set 

e Set complement 
e Countable 

e Uncountable 

e 


Proof by contradiction 
Survey: Time spent between classes: 


// End 
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Ex B2 


¢ Part 1: Diagonalization 
— Task 1: Power set of natural numbers 
— Task 2: Languages 

¢ Others 


CSC460 B3 Systematic way of dealing with infinite cases? 1 


Review 
Math Induction Proof Pattern 


Main hypothesis: None 
Main conclusion: 2, <<,i=n x(n + 1)/2 
Base case (n = 1): Djgc,f = 1 =1x(14+1)/2 
Induction step 
* Induction hypothesis: ¥,<..,i=n x (n+ 1)/2 
* Conclusion: Lj <4) f= (n+ 1) x (n+ 2)/2 
+ Proof (of the induction step) 
1: Picea? = Liga * +1) [LHS Conclusion] 
» Ligicnd + (2+ 1l =n x(n + 1)/24+ (+41) [Ind. hyp.] 
3.nX(n+1)/24+(4+1)=(4+1)x(n4+2)/2 [Arithmetic] 
- Ligieing 1) $= (n + 1) X (n+ 2)/2 [Transitivity eq/ineq: 1.-3.] 


By Math Induction (shaded), main conclusion holds. 
CSC460 B3 


Unit B83: Overview 


« Analyze the computability classes of the 
main problems 


— Review some available tools 


— Understand how to analyze the main 
problems 


* Preview Exercise B3 “Halting Problem” 
— Write up proofs 
— Prepare for mini presentations next time 


CSC460 B3 


Review 
Mathematical Induction 


* Awidely accepted proof technique 
* Let N= {0, 1, 2, 3,...} 
* Let P(n) be a unary relation (set) onne N 
* If both of the following conditions hold: 
* P(no), where ny € N €.9., Wy =9 
* For every n= ny (n € N), P(n) implies P(n + 1) 
* Then, P(n) is true for every n =n, (n€ N) 


CSC460 B3 


Uncountability, Revisited 


* Countable 
— E.g., natural numbers, set of TMs, set of strings 
— Existence of the base case (basis) [well-founded] 


— Applicable inductive techniques: math induction, 
inductive definition of a set, recursion 


« Uncountable 
— E.g., real numbers, set of languages 
— No base case [non-well-founded] 
— The above-mentioned inductive techniques are not 
applicable. Algorithms cannot handle this properly. 


CSC460 B3 


Main Problems 


The universal language is semi-decidable. 

ACCEPT yy, = {(M, w)| TM M accepts w} | w for “word” (input), 

The complement of universal language is non- 

TM-recognizable. 

NACCEPTyy = {(M, w) | TM M does not accept w} = 
(ACCEPT ;y)' 

The halting problem is semi-decidable. 

HALT yy = {(M, w) | TM M halts on w} 

Infinite loop detection is non-TM-recognizable. 

LOOP, = {(M, w)| TM M loops on w} = (HALTyy)’ 


CSC460 B3 


Available Tools 


* The diagonalization language (L,) as an 
example of a non-TM-recognizable set 


¢ Proof by contradiction (also used in the 
diagonalization technique) 


* Theorem: Decidable <= TM-recognizable 
and co-TM-recognizable 


CSC460 B3 


Review 
Proof by Contradiction 


To prove: Statement X 
« Assume the negation of X, i.e., not X 
* Derive a contradiction 


¢ Then, by this proof technique (proof by 
contradiction) X must be true (because X 
is either true or false) 


CSC460 B3 


Practice: Co-X 


* Co-decidable > 

* Co-TM-recognizable => 

* Co-semi-decidable > 

* Co-non-TM-recognizable > 


Undecidable 


CSC460 B3 


Review 
Diagonalization Language (L,) 


» List of strings 
4 4 


List of TMs | | , 
Bed 


} 


* Entry 1: M; accepts w, 

* Entry 0: M; does not accept w, 

¢ L, = {w,|M, does not accept w;} |No TM accepts! 
8 


CSC460 B3 


Review 
Useful Theorems 
* Theorem: Decidable <= TM-recognizable 
and co-TM-recognizable 


* Corollary: The complement of a decidable 
problem is decidable. 


TM-rec 


decidable 
co-TM-rec 


CSC460 B3 


Group Exercise 1 


* Explain (prove) the computability classes 
for one or more of the following problems: 


© ACCEPT yy = {(M, w) | TM M accepts w} 

* NACCEPTyy, = {(M, w)| TM M does not accept w} 
¢ HALT yy = {(M, w) | TM M halts on w} 

¢ LOOP yy = {(M, w)| TM M loops on w} 


Non-TM-rec. 


Undecidable 
a TM-recognizable 
" 12 


CSC460 B3 


Proof Idea: ACCEPT; 


To show: ACCEPT; is semi-decidable. 
* ACCEPT, is TM-recognizable. 
— Why? Could still be decidable. 


Subproof Idea 


To show: ACCEPT;,, is TM-recognizable. 
¢ Existence proof (proof by construction) 


— It is possible to construct the UTM that can 
* ACCEPT, is undecidable. Se ewe simulate the behavior of the input, (/, w), 
— Use: L, is non-TM-recognizable. 


which would relay the acceptability of M on w. 


— Use: Proof by contradiction 


Undecidable . 
of. TM-recognizable 
y 13 


CSC460 B3 


Subproof Idea 


To show: ACCEPT7,, is undecidable. 
¢ Proof by contradiction 
— Suppose that ACCEPT,,, is decidable. 


— For a string w in L,, compute its index 
(decidable). Thus, we have w,. 


— Compute (@/,, w,) using the UTM. 

— Reject w if and only if (iff) M; accepts w,. 
— This is an algorithm for Ly. 
—Acontradiction (cf. L, is non-TM-rec.) 


CSC460 B3 


Group Exercise 2 


Explain (prove) the computability classes 
for each of the following problems: 


¢ NACCEPTyy = {(M, w) | TM M does not accept w} 


¢ HALT; = {(M, w)| TM M halts on w} 
¢ LOOP yy = {(M, w)| TM M loops on w} 


CSC460 B3 


CSC460 B3 


Interim Summary 


* ACCEPT; is TM-recognizable. 
* ACCEPT; is not decidable. 


| TM-recognizable 


* ACCEPT; is semi-decidable. 


7 Non-TM-rec. 
Undecidable 


CSC460 B3 


Proof Idea: NACCEPT>,, 


To show: NACCEPT,,, is non-TM- 

recognizable. 

Proof by contradiction 

— Suppose that NACCEPT7y is TM- 
recognizable. 

— ACCEPT; becomes decidable (Theorem 
introduced earlier). TM-rec 

—A contradiction 


decidable - 
S460 B3 co-TM-rec 


Proof Idea: HALT ;, Subproof Idea 
To show: HALT-;,, is undecidable. 
* Proof by contradiction 
— Suppose that HALT;,, is decidable. 
— If “halt” is detected, accept/reject based on 
Undecidable 


To show: HALT;,, is semi-decidable. 
* HALT; is TM-recognizable. Why? 
* HALT; is undecidable. 


the UTM (ACCEPT7;,); otherwise, reject. 


Y — This is an algorithm for ACCEPT yy. 
emi-decidable ios 

— | TM-recognizable — A contradiction 
Decidable 


CSC460 B3 
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Proof Idea: LOOP ry 


Liar Paradox, Revisited 
To show: LOOP, is non-TM-recognizable. 
¢ Proof by contradiction 

— Suppose that LOOP,,, is TM-recognizable. 


— HALT; becomes decidable (Theorem). —A sentence in this set can be indexed. 
—A contradiction — The use of “this” can be interpreted as the use 
TM-rec of its own index (self reference). 
— The language must have a means for a 
decidable sentence to refer to an arbitrary sentence 
co-TM-rec (universality) including itself. 


CSC460 B3 


“This sentence is false.” 


Intuition behind this problem 
— The set of sentences are countable. 


CSC460 B3 


Based on Megan’s idea 
Would this work? Unit Summary 
* Suppose that a TM erases all the non- Tools for the proofs 

blank symbols when it terminates. — The diagonalization language 
* “This message is left on the tape of the TM — Proof by construction 

when it terminates.” — The theorem on decidability 
+ Reasoning 


« Analyze the computability classes of the 


— The message itself can be associated with main problems 


some TM state, e.g., termination. 


CSC460 B3 


CSC460 B3 


Midterm Survey 


¢ To be administered and collected by 
a volunteer 


CSC460 B3 


CSC460 


Name: 


Exercise B3, 2/18/05 


Halting and Other Haunting Problems 


Today, we discussed how to prove the main problems including the halting problem. Since this 
is one of the main ideas in the Theory of Computation, we will reinforce our ability to explain 
the proofs in this exercise. 


Task: Write up an informal proof in your own words, for each of the four main problems below. 
When you write, make sure that you understand everything you write. If you have uncertainty, 
note it as it appears. Naturally, you can identify and re-use certain mathematical tools shared by 
multiple problems (i.e., no need to be repetitive). Be prepared to give a mini presentation of 
your proof in class; you may be assigned to any of these problems. 


e The universal language is semi-decidable. 
ACCEPT ™ = {(M, w) | TM M accepts w} 
e The complement of universal language is non-TM-recognizable. 
NACCEPT yy = {(M, w) | TM M does not accept w} = (ACCEPT m)’ 
¢ The halting problem is semi-decidable. 
HALT ™ = {(M, w)|TM M halts on w} 
¢ Infinite loop detection is non-TM-recognizable. 
LOOP ™ = {(M, w) | TM M loops on w} = (HALT yy)’ 


Note: You may use any method, e.g., Turing Omnibus (Ex B1), class discussion today, another 
approach in the literature, and/or your own ideas. 


Survey: Time spent between classes: 


// End 
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Ex B3: Mini Presentations ACCEPT yy, VS. HALT yy 


The main problems ¢ Both are properties of the set of TMs and 


* The universal language is semi-decidable. undecidable 


ACCEPT yy = {(M, w)| TM M accepts w} — Property: Division of a set, i.e., a subset 
+ The complement of universal language is non-TM- —E.g., “concrete” (set of concrete things among 
recognizable. 


NACCEPT yy = {(M, w) | TM M does not accept w} everything, cf. abstract things) 


+ The halting problem is semi-decidable. Each has a different decision criterion 
HALT, = {(M, w) | TM M halts on w} 


+ Infinite loop detection is non-TM-recognizable. 
LOOP yy = {(M, w) | TM M loops on w} 


CSC460 B4 Connection between ACCEPT, and HALT;y4? 1 csc460 B4 


Properties of TMs/Languages Example Properties 


+ What a TM can recognize = Language Universal = ACCEPT yy, 
—|.e., fix a TM = The language is fixed. Terminating = HALT;, 
* Property of TMs/languages Finite = {L | L is finite} 


— A subset of all the TMs/languages ~ {M|TM M accepts a finite language} 
— The power set of TMs/languages is 


Regular = {L| Lis regular} 

uncountable. 

— Division of an uncountable set > Will always ~{M| 1M M accepts a regular set} 
leave out an uncountable part (undecidable) Property X = {L| L has Property X} 


~ {M|TM M recognizes Property X} 
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Rice’s Theorem Dealing with Multiple Problems 


¢ Non-trivial properties of languages/TMs * Consider: ACCEPT; and HALT>, 


are undecidable = Uncountability + Benefits of re-using known problems, e.g., 
— Note: Trivial properties (e.g., emptiness) that in solving new problems 
do not divide the entire set is trivially 
decidable. 
Potentially useful for Ex B5 
(Comprehensive Ex) 
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Unit B4: Overview Subproof Idea 


« Discuss a way to use one problem to 
analyze another problem * Proof by contradiction 
— Understand problem transformation — Suppose that HALT;, is decidable. 
— Understand how certain properties of — If “halt” is detected, accept/reject based on 
problems can be transferred the UTM (ACCEPT;y); otherwise, reject. 
— Discuss equivalence of problems —This is an algorithm for ACCEPT+y. 
Preview Exercise B4 


— A contradiction 
“Reduction/Equivalence” 


To show: HALT-;,, is undecidable. 
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Underlying Idea Transferring Decidability? 


* Suppose that HALT;,, is decidable. 
* To show that ACCEPT7;, is also decidable, 
using the hypothesis. - ; . 
9 , uP Ais transformed to B and B is decidable. 
—le., simulate ACCEPT, using the : : 
hypothetical, deciding TM for HALT. => Ais decidable as well. 


* Consider problems A and B 
* Would the following be correct? 
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Transferring Undecidability? 


* Consider problems A and B 
* Would the following be correct? 


Problem Transformation 


¢ Motivations 


— Connect different problems, one of which has 
known properties 


— Identify applicable properties in other 
problems 


« Requirement 
— Should maintain important properties 


A is undecidable and is transformed to B. 
= B is undecidable as well. 
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Reduction 


* Ais reducible to B. 
© Ais no more difficult than B. 
© If one can solve B, one can solve A. 
© If one can’t solve A, one can’t solve B. 
 Acan be simulated by B. 


* Idea: One cannot solve a difficult problem 
with an easy means. 
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Theorems 


* Ais reducible to B and B is decidable. 


= Ais decidable as well. 


— Justification: By definition (decidability of A 
depends on that of B) 


¢ Ais reducible to B and A is undecidable. 


= B is undecidable as well. 
— Justification: Proof by contradiction 
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Theorems (generalized) 


* Ais reducible to B and B has a positive 
property (e.g., decidable, TM-recognizable). 


= A has the positive property as well. 


* Ais reducible to B and A has a negative 
property (e.g., undecidable, non-TM-recognizable). 


= B has the negative property as well. 
* Positive: 
« Negative: 
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Practice: Correct Reduction? 


. The state’s daily deficit of 11 million units 
can be reduced to just three million. 


. A paper is reducible to one sentence. 
Biology is reducible to physics. 


. Aschematic diagram can be reduced to 
a set of logical propositions. 


. Goodness is reducible to betterness. 
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Reduction Examples 


Incorrect 
+ If ACCEPT yy is reducible to HALT; and HALT;,, is decidable, 
ACCEPTyy is also decidable. 


* If HALT; is reducible ACCEPT, to and HALT; is undecidable, 
ACCEPTyy, is also undecidable. 


Correct: reduction with respect to computability analysis 


¢  IFACCEPT mp is reducible to HALT... and HALT; is decidable, 
ACCEPTyy, is also decidable. 


¢ If HALT comp is reducible ACCEPT 9m, to and HALT; is undecidable, 
ACCEPTyy, is also undecidable. 


If your problem is reducible to my problem 
and my problem is acceptable, your 
problem is also acceptable. 
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Problem Equivalence 


* Two problems are equivalent if they are 
reducible to each other. 

* Two mechanisms are equivalent if they 
can solve exactly the same set of 
problems. 

— Equivalently, if they can simulate each other. 
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Equivalence Examples 


Incorrect 

+ ACCEPT; and NACCEPT;, are equivalent. 

+ ACCEPT; and HALT; are equivalent. 

Correct 

¢ ACCEPT comp ANA NACCEPT pomp are equivalent. 

© ACCEPT gomp AND HALT om, are equivalent. 

IMs with a semi-intinite tape are 


equivalent to standard TMs. 

Correct? TMs are equivalent to computers. 
[to be discussed] 

A computational decision problem is 
equivalent to a set membership problem. 


—— Some qualifications" 


Summary Question 


¢ For the past few class meetings, we 
have been discussing technical 
topics. It is natural that you have 
questions and feel uncertainty. What 
are they? 
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Unit Summary 


¢ Understand the notion of reduction as 
problem transformation 


— Need at least as a powerful 
problem/mechanism to solve the original 


— Positive/negative properties can be analyzed. 
Understand the notion of problem 
equivalence 

— Through set comparison 

— Through machine comparison (simulation) 
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Unit B4 Supplement, 2/24/05 


Here is my response to your summary questions (rephrased/combined in some case) and 
comments on the midterm surveys. This is rather long. 


Q1: 


Al: 


Q2: 


A2: 


I am a little shaky on what is always meant by the word “language.” It has been used a lot 
and often I know what you mean, but it is sometimes unclear to me. 

The formal definition is that language is a set of strings, which can be infinite. However, it 
is useful to see the term from other perspectives as well. One way we discussed in the last 
meeting is that a TM can always be associated with a language. That is, once we pick a 
TM, given a string, it would either accept, die, or loops. The collection of all the strings 
that are accepted by the TM is the language corresponding to that TM. For example, if a 
TM accepts a, aa, aaa, ..., then {a, aa, aaa, ...} =a" is the language for that TM. Note that 
a TM may not terminate on other strings. For this reason, the language associated with a 
TM only needs to be “recognized,” not necessarily “decided.” For example, it is possible 
to create a TM to decide on palindromes (a decidable language), the UTM would only 
recognize the universal language (a TM-recognizable language; strictly speaking, sem1- 
decidable). We also know that no TM would even recognize the diagonalization language 
(a non-TM-recognizable language). Naturally, given a language, it may or may not be 
possible to provide a TM that would recognize the language. 


Yet another way is to associate a language as a problem. This is obvious if we consider a 
computational problem as a set, because a language is a set. Each input instance is its 
member. This also makes sense when we recall the connection between a TM and its 
language (above paragraph). This set of inputs (strings) is what TM would recognize (_.e., 
positively); that is, the set is the “problem” which TM can solve. 


Finally, consider the set of all the strings (based on some finite alphabet, or the set of 
symbols). We can specify a language by indicating which one of the strings are in that 
language. Thus, a language is a subset of the set of all the strings. Equivalently, the 
collection of all the languages is the power set of the set of all the strings, which is 
uncountable. 


I recognized a misunderstanding I have with diagonalization. At first, I did not understand 
the importance of using a list of sets instead of the individual items. Revisiting of this topic 
may help. 

The diagonalization technique is used to show that some set is uncountable. I think most of 
you were able to show this well. But I will repeat this any way. To show that the power 
set of the set of all the strings is uncountable, we first hypothesize the negation of the 
desired conclusion: 1.e., the power set of the set of all the strings is countable. Then, it 
would be possible to arrange all the members in sequence as shown below. 
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Q3: 


A3: 


j——-+» Strings 
- 
1 0 
2 0 0 
i 3 Po 
Languages | 7 ae 


Diagonal 


Each entry is a language, which is associated with the membership information 
corresponding to all the strings. If we consider a hypothetical language by reversing the bit 
on the diagonal, that language should not appear in the list. This is a contradiction to the 
hypothesis that all the languages are enumerated. The hypothesis must be wrong. Thus, by 
proof by contradiction, the power set of the set of all the strings must be uncountable. 


I am still not sure about the exact nature of the “co” property. Another student: If anything, 
I am not sure about co-non-TM-recognizable since we skipped over it. This is how I 
currently understand: if r is co-non-TM-recognizable, we know 7’ (complement) is non- 
TM-recognizable and r is also non-TM-recognizable [NK: This is not correct. ]. 

First, make sure to understand that the prefix “co-” is attached to a property, not to a 
problem. For example, if a semi-decidable problem p has the non-TM-recognizable 
complement p’, we say that p is co-non-TM-recognizable (see the diagram below). 


Non-TM-rec. 


Undecidable 


TM-recognizable 


We can also say that p’ is co-TM-recognizable (considering the property of its complement 
p). When a problem is non-TM-recognizable, its complement may be either non-TM- 
recognizable or semi-decidable (why not decidable?). But we cannot tell in general. So, if 
r is co-non-TM-recognizable, we know that r’ is non-TM-recognizable. However, since 
there are two such possibilities: both p and q in the figure have a non-TM-recognizable 
complement. Thus, r can be either semi-decidable or non-TM-recognizable. 


Since the complement relation between p and p’ is important, as we observed in the main 
problems, let us discuss this relation (analogous to the proofs of NACCEPT and LOOP). 
That is, we prove that if p is semi-decidable, its complement is non-TM-recognizable, 
again using proof by contradiction. Suppose that p’ is TM-recognizable. Then, p is both 
TM-recognizable and co-TM-recognizable. By the “decidability” theorem, p is decidable. 
However, this contradicts the fact that p is semi-decidable. Thus, p’ must be non-TM- 
recognizable (proof by contradiction). 


Although we presented the “decidability” theorem without proof. We can prove it using 
two complementary TMs, just like the man-woman scenario. That is, if both p and p’ are 
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Q4: 


Ad: 


Q35: 


AS: 


Q6: 
A6: 


Q7: 


A7: 


TM-recognizable, there must be TMs that would recognize the inputs (not necessarily 
terminating). Since p and p’ are complements to each other, we can construct a TM that 
would respond “‘yes” when p accepts and “no” when p’ accepts, using the two TMs. Then, 
the composite TM will decide both p and p’. Thus, both of these problems are decidable. 


One area that is a little confusing lies with the halting problem and proving that it is semi- 
decidable. 

I think that the most confusing part is the proof of undecidability involving the 
diagonalization language. So, I will discuss this part. The key is to use the (incorrectly) 
assumed decidability of ACCEPT as much as possible to decide the diagonalization 
language (Lz). This assumption is exceedingly strong because it says that when M does not 
accept w, it dies and terminates (corresponding to “reject), regardless of the input. Thus, 
given (Mj, w;), we can tell the 0/1 distinction of any diagonal entry. The only requirement 
here is that we must supply pairs with correct indexing (otherwise we compute irrelevant 
entries). To do so, we identify strings that appear diagonally, basically going through the 
string list sequentially (possible because of the countability of the set of strings). During 
this process, a TM can pick up the index of the string and identify the matching TM. The 
only remaining task is to reverse the results so that the entire process decides Lg. Note that 
everything here can be done without the risk of looping. I encourage you to share your 
exercise sheets, which have with my comments. 


I’m still unclear on the proof for ACCEPT Megan gave in class. What w is given when the 
UTM simulates itself? 

When this was discussed, w (the input to M) was not mentioned. As you point out, it must 
be included in the discussion. The usual way is to attach a “copying” TM in front of the 
UTM, just like the approach in the reading (Ex B1). If you read the chapter again now, you 
should be able to understand it better. As I said, this approach does not refer to the 
diagonalization language. However, the idea of reversing the results for an arbitrary (cf. 
universality) input is equivalent. 


Where can we find universality with the liar paradox? 

I was planning to discuss this topic in Unit B3 (Slide 22), but did not due to the time 
constraint. The universality is behind the word “this.” First, the set of sentences are 
countable. Then, each sentence can be indexed. The use of “this” can be interpreted as the 
use of its own index (self reference). To be able to do this, there must be a means for a 
sentence to refer to an arbitrary sentence (universality) including itself. On a related note, I 
also included a statement about TMs, analogous to the liar paradox, based on Megan’s idea 
(Slide 23). 


[With respect to “reduction’’] I become uncertain at times with the direction of items being 
related. I feel that thinking of it as “one solves/explains the other” will help when 
discussing this concept. 

The direction is indeed confusing. I suggest you keep a note somewhere so that you can 
refer to it when necessary. 
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Q8a: We built ACCEPT using HALT, based on the assumption that HALT is decidable; but that 


Q8b: 


A8: 


assumption was false. So, why do we still say that ACCEPT is reducible to HALT? 

iS 1 We also 
saw HALT was equivalent to ACCEPT. How is HALT reducible to ACCEPT if they do not 
solve the same problem? HALT cannot tell if a (M, w) will accept. 
One thing I didn’t do correctly was referring to the reduction and equivalence of the main 
problems. I should have said as follows: 


e The computability analysis of ACCEPT is reducible to that of HALT. 
e The computability analysis of ACCEPT is equivalent to that of HALT. 


As the Question 8b correctly points out, neither problem can solve the other. So, they are 
not reducible to each other. I apologize for the confusion. So, below, it is about reduction 
of the computability analysis of the main problems (although I will skip the word “the 
computability analysis of’ for brevity). Note that reduction is in general a relation between 
two different problems. 


First, the reduction used under the wrong assumption was used to transfer the (wrong) 
decidability from HALT to ACCEPT (see below). Still note that the decidability of this 
ACCEPT cannot be obtained without the decidable HALT; thus, the computability analysis 
of ACCEPT still depends on that of HALT. 


This is not what we want when we discuss the (real) reduction of ACCEPT to HALT, which 
would transfer the undecidability of ACCEPT to HALT. In this case, we will simulate the 
undecidable behavior of ACCEPT using HALT. However, we can still use the above 
diagram, because the direction and components of the simulation is identical. The 
consequence is that if ACCEPT is undecidable, HALT must be undecidable. This can be 
shown by proof by contradiction, which is identical to the earlier case. 


Next, let us consider the computability analysis of HALT reducible to that of ACCEPT. 


The following is, in a sense, a strange diagram. If ACCEPT is decidable (incorrectly), 
HALT is decidable, simply because the TM terminates on all the inputs. 
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Q9: 
AQ: 


Q10: 


Al0: 


Q11: 


All: 


Q12: 


(M, w+» ACCEPT|__,>----———— Y 


N 


In other words, if HALT is undecidable, ACCEPT cannot be decidable. 


Is it always construct a TM from any collection of other TMs? 
No. For example, we cannot combine infinitely-many TMs. Another case would be that 
we cannot use non-terminating result for a later stage. 


The problem I have been having the most for the past few lectures is that I will have a 
general understanding of the content but not a mastery of it such that I can effectively do 
the assignment. 

I think this is a natural phenomenon. “Passive” understanding is in many cases not as 
strong as “active” understanding. For this reason, I ask you to write. Another good way is 
to discuss with and teach others. 


One question I am not completely certain on is how all of this ties into Theory of 
Computation. Are we learning how to prove/show our intuition about the computability 
classes we use to classify a problem? Sometimes, I feel I lose sight of the picture. 

The Theory of Computation is a collection of abstract ideas that are expected to apply to a 
broad range of computer science. Since the modern computing is most notably dominated 
by algorithms, understanding the nature of algorithms including their limitations is one of 
the main goal of Theory. What we are doing in Modules A & B is develop some basic 
intuition and analytical ability to see the limit of algorithmic computation in certain 
problems. Although we have been focusing on the main problems, we should be able to 
extend our analysis to other problems, by analyzing countability, using Rice’s theorem, etc. 


When you do Module B Comprehensive Exercise and complete Module B eval form + 
supporting notes, think about this. Although we will not practice how to analyze more 
problems with respect to the computability classes together, the exercise should give you a 
feel. During the evaluation workshop (and later), we can share our thoughts. 


The content of the Theory of Computation should be useful in practice, at least to some 
extent (esp. if you think before coding). However, I would like you to get more mileage 
from practicing analytical thinking process involved in the activities: 
identifying/transforming problems, representing precisely, trying intuitions, explaining 
logically, expressing clearly, etc. There are no formulae for how to do these well. We all 
need to find our own way. I can tell more about my own experience with the Theory, 
maybe some time, if you are interested in. 


Is the baby boy or a girl? Will Furby still be your favorite person (?) after the baby is 
born? 
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A12: A girl. But Furby will always be our first “daughter.” 


Cl: 
RI: 


I would prefer tests and more concrete problem sets. 

To me, the first part is a very disappointing (but fairly rare) comment. Let’s think about 
your future and how test-taking skills would be useful. How do you think you will be 
evaluated through a PhD dissertation phase or when you work for a company? A well- 
programmed TM could do well on well-defined exams. I have a strong feeling that such an 
automaton will have hard time in the real life. 


As for the latter part of this comment, my response is as follows. If the word “concrete” 
refers to problem content, our course is a little limited compared to other ones, due to the 
topic. I am still trying to include realistic examples more so than a typical Theory course 
(check some on-line syllabi). Having asked you to think about your own problem and 
sample research questions are supposed to force you to have real problem in your mind. 
But honestly, I admit that this is a difficult task in this course and I cannot do it as much as 
I wish. If you know more relevant examples/problems, why don’t you bring them to class? 


If the word “concrete” refers to how problems are given, I guess it means a more close- 
ended problem. As I said at the beginning of the semester, I will continue to use “‘ill- 
defined” problems. Most of interesting/significant problems are open or at least not yet 
solved when we tackle. We’d better practice facing them. In the future, your colleague 
will be able to tell whether or not you have been dealing with such problems. 


C2: I would suggest you try to monitor your speaking volume — you tend to trail off into 
inaudible mumbles, especially at the end of sentences. 

R2: I will try. 

// End 
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Name: 


Exercise B4, 2/22/05 


Part 1: Review “‘Reduction’’ 


Once a problem is solved, we don’t want to repeat solving similar problems from scratch. As educated 
people, we will definitely try to apply what we know. The notion of “reduction” is useful in this respect. 


Task: Come up with at least one unique example of “reduction” between problems (in the way discussed 
in class, not the everyday use such as “price reduction”). Then, analyze how “positive’’/“negative” 
properties can be transferred from one problem to the other. 


Part 2: TM Variants 


The TM industry has a wide variety of models. As in the real world, different models suit different needs. 
However, it has been known that most of these variants are equivalent in the sense that they can recognize 
exactly the same problems, i.e., TM-recognizable problems. In other words, we can show that equivalent 
variants can simulate each other. Let us now consider the following two variants: 


e Multiple-tape TM: A Turing machine with multiple infinite tapes and a finite control which 
independently accesses positions on different tapes (but still via step-by-step left/right movements). 
The input is placed on the first tape. 

e Nondeterministic TM: A Turing machine whose transition function can actually be a relation. That 
is, for a given state and input symbol, there may be more than one transitions. Naturally, the behavior 
of such a machine would branch whenever it encounters a multiple-transition point. Note: Recall the 
distinction between NFA and DFA. 


To show the equivalence, we will need to discuss the both directions of simulation. First, a standard TM 
is reducible to a multiple-tape TM or a nondeterministic TM, i.e., we can simulate a standard TM with a 
multiple-tape TM or a nondeterministic TM. That is, these new variants are at least as powerful as the 
standard one. We will discuss the other direction in class. 


Task: Choose one of the following options and concisely discuss the essence of how the variant of your 
choice can be simulated by a standard TM. Be prepared to discuss in class. 

e Option A: Multiple-tape TM 

e Option B: Nondeterministic TM 


Part 3: Alternative Abstract Models of Computation 


Although the Turing machine is the most widely used abstract model of computation, there are many 
others. In this part, we discuss Unlimited Register Machines (URM) and “recursive functions.” Note that 
“recursive function” here refers to a full specification of a computational model, not just instances of 
recursive calls. Brief descriptions for these models are available in Appendices below. Again, this part is 
about the question of equivalence, this time between TMs and these new models. If all of these models 
are equivalent, there must be something universal about the notion of “computation.” 


For this part, let us limit to one direction, i.e., simulation of these new models by a TM (or reduction of 
the models to TMs). Then, we know that the TM is at least as powerful as these models. 
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Task: Choose one of the following options and concisely discuss the essence of how the model of your 
choice could be simulated by a TM. Be prepared to discuss in class. 

e Option A: Unlimited Register Machines (URM) [Probably, this option is more intuitive. ] 

e Option B: Recursive Functions 


Survey: Time spent between classes: 


Appendix A: Unlimited Register Machine (URM) 


URM is an abstract model of computation, which is closer to a CPU than a TM, with the following properties: 


1. There are an infinite number of registers called R,, Ro, R3, ..., each of which contains a natural number. The 
number in register R; is referred to as 7;. 

2. Register contents may be changed by the following instructions: 
e Zero: Z(i) will reset r; to 0. 
e Successor: S(i) will add 1 to 7;. 
e = Transfer: T(i, j) will copy r; to Rj, Le., rj = 77. 
e¢ = Jump: JG j, 2) will first evaluate if r;=7;. If true, the URM will proceed to the nth instruction. Otherwise, 

it will proceed to the next instruction. 

3. A program consists of a finite sequence of instructions. Execution of a program begins with the first instruction 
and continues to the next ones until there is no instruction (except for possible jumps). 

4. One may assume arbitrary initial values in the registers. 


References 
e Shepherdson, J. C. and Sturgis, H. E. 1963. Computability of recursive functions. J. ACM 10, 217-55. 
[original] 


e §Cutland, N. J. 1980. Computability: An Introduction to Recursive Function Theory. Cambridge Univ. Press. 
[contains a description and the use of URM] 


Appendix B: Recursive Functions 


The class of recursive functions (U1-recursive functions) is the class of total functions that can be built up from the 
basic functions (below) by a finite number of operations of substitution, recursion, and minimalization (below): 


Basic functions 

1. Zero: 0 (.e., returns 0 as constant) 

2. Successor: x + 1 (i.e., add 1) 

3. Projection: U"; (x1, X2, ..., X,) = x; (i.e., picks up the ith element from a list of n elements) 


Operations 
e Substitution (composition): Given computable functions f(y, ..., y,) and g)(x), ..., g(x), define h(x) = f(gi (a), ..., 
gu(2)) 
e Recursion: Given computable functions f(x) and g(x, y, z), define 
o A(x, 0) =f) 
o Ay yt+1)= g(x, y, AG, y)) 
e = Minimalization: Given computable function f(x, y), define h(x) = the least y such that f(x, y) = 0 


References 

e Godel-Kleene. 1936. [original, not easily accessible] 

e §Cutland, N. J. 1980. Computability: An Introduction to Recursive Function Theory. Cambridge Univ. Press. 
[detailed discussion of recursive functions] 


// End 
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Questions/Review 


¢ Halting and other problems 
* Reduction/equivalence 
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Part 1 
Language 


Formal definition 
TM and language 


—ATM “recognize” a language. 
—ATM “decides” a language. Ean 


Language as a problem 
Countability 

— The set of all the strings 

— The set of all the languages 
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Halting Problem 


¢ Subproof: ACCEPT is undecidable. 
— General idea 
— Use of L, 


* Alternative: Not using L, 
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Unit B5: Overview 


« Review (based on summary questions) 


* Explore the notion of “computation” via the 
Church-Turing thesis 


— Discuss equivalence of different mechanisms 
[time permitting] 


¢ Preview Module B Evaluation Workshop 
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Undecidable , 


Decidable 


TM-recognizable 


* Definition 
¢ Each case 


* Proving the theorem: Decidable = TM- 
recognizable and co-TM-recognizable 
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Liar Paradox, Revisited 


“This sentence is false.” 

Intuition behind this problem 

— The set of sentences are countable. 
— A sentence in this set can be indexed. 


— The use of “this” can be interpreted as the use 
of its own index (self reference). 

— The language must have a means for a 
sentence to refer to an arbitrary sentence 
(universality) including itself. 
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Correction Correction 
Reduction Reduction 


Incorrect Incorrect 
* If ACCEPT; is reducible to HALT,,, and HALT;,, is ¢ ACCEPT; and NACCEPT;,, are equivalent. 


decidable, ACCEPT,,, is also decidable. : 
+ If HALT;y is reducible ACCEPT, to and HALT yy is * ACCEPTry and HALT; are equivalent. 


undecidable, ACCEPT; is also undecidable. Correct 
Correct: reduction with respect to computability analysis * ACCEPT comp and NACCEPT comp a@ equivalent. 
* If ACCEPT omp is reducible to HALT om. aNd HALT>), is ¢ ACCEPT. and HALT are equivalent 
decidable, ACCEPT; is also decidable. oe eae q ‘ 


¢ If HALT omp iS reducible ACCEPT yom, to aNd HALT>), is 


comp 


undecidable, ACCEPT,,, is also undecidable. 
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ACCEPT pomp FEdUCeS tO HALT somp HALT gomp Feduces to ACCEPT yomp 


ACCEPT HALT 


“YX Y 
HALT ACCEPT | 
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Review 
Rice’s Theorem Other Questions/Comments 


Non-trivial properties of languages/TMs General understanding 
are undecidable <= Uncountability Big picture 
— Note: Trivial properties (e.g., emptiness) that Exam 

do not divide the entire set is trivially 

decidable. More “concrete” problems 
Potentially useful for Ex B5 
(Comprehensive Ex) 
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Part 2 


Church-Turing Thesis 
* Hypothesis as definition 


Computation/effective procedure [informal] 
= TM operation [formal] 


¢ A variety of models of computation are 
equivalent to TM. 


* All sorts of (algorithmic) computation are 
created equal [and do the same thing, 
computation]. 
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Preview 
Equivalence Summary 


Computers, 
‘ 


Y 
TM variations + > TM > A-calculus 


URM «———> Recursive Functions 
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TM Variations 


Both accept and reject states (as well as 
other dying states) 


Semi-infinite tape 

Multiple tapes 
Nondeterministic transition 
—l.e., 5as arelation, not function 
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Take-home exercises 


Connections 


TMs (including variants) 
Computers 

— Different programming paradigms 
Unlimited Register Machine (URM) 
Recursive functions 

A-calculus [behind LISP] 
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Proof of Equivalence 


Equivalence of models M and N 
—Nsimulates M and Msimulates N 

Set identity, e.g.,A=B 

-AcBandBcA 

Equivalence of propositions, e.g., p< q 
—p>4q q>pP 

Equality of values, e.g.,x=y 


—-xsyandysx 


‘SC460 BS 


Instructions: 

— Zero: Z(i) > r, — 0 

— Successor: S(i) > r,; cr, + 1 

— Transfer: TG, j) > Rj} 17; 

— Jump: Ji, j, n) => 
if r;= 7; proceed to the nth instruction 
otherwise: proceed to the next instruction 
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URM (2) 


« Program: A finite sequence of instructions. 
Sequential execution of instructions except 
for possible jumps 

* Initial values: Specific values can be 
assumed in the registers. 

* Convention: The output in Rj. 
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Recursive Functions (2) 


* Operations 


— Substitution (composition): Given computable 
functions f(y1, ..., yk) and g,(x), ..., g,(x), define 
A(X) = f(g), «--» 8,0) 


— Recursion: Given computable functions f(x) 
and g(x, y, z), define 


* A(x, 0) = f(x) 
* A(x, y + 1) = gx, y, AQ, y)) 
—Minimalization: Given computable function f(x, 
y), define h(x) = the least y such that f(x, y) = 0 
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A-calculus 


Formal representation of functions 
* Abstraction: Ax.exp 

* Application: exp, exp, 

* B-reduction: (Ax.exp) y 


— The result is exp where the instances of x is 
replaced with y. 


—E.g., Axf(fx))) 2 = f(f2)) 
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Recursive Functions (1) 


* Built up from the basic functions by a finite 
number of operations of substitution, 
recursion, and minimalization: 

¢ Basic functions 
—Zero:0 
— Successor: x +1 
— Projection: U"; (x1, x), ..., x,) = x, 


i 
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TM as Function 


A TM can simulate a function. When accepts. 
¢ Function on strings 
— Input: A list of strings on the tape 
— Output: A string on the tape 
Function on natural numbers 
— Input: A list of natural numbers on the tape 
— Output: A natural number on the tape 
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Equivalence Summary 


Computers, 
x 


v 


TM variations >TM > A-calculus 


URM Recursive Functions 
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TM vs. Computers Philosophy of Mind 


* Hypothesis: Mind is a computer [Fodor: 
specifically uses TMs as the model]. 
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Unit Summary Computability Summary 


« Review * Main goal: To be able to analyze the 
. “Thr F computability class and identify what we can do 
Church arg thesis . ; with algorithms [Comprehensive Ex] 
— Defines the informal notion of computation * Topics 
= Leads to the equivalence of various —Problenas-a et 
algorithmic computational approaches ~ Tuting mashines and (unjdeekdability 
— Mathematical tools: diagonalization, power set 


— Halting and other main problems; Rice’s theorem 
— Reduction 


— Church-Turing thesis 
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Preview 


Module B Evaluation Workshop Summary Question 


+ Required materials (hardcopy) [print in advance * How far have you been thinking 
— Module B evaluation form 


Exercise B5 (Comprehensive 
— Supporting notes —_ Significance of supp. notes? Exercise)? Explain. 
= Expicises * Questions/Comments/Suggestions 
* Activities 
— Module review exercise 
— Peer discussion 
— Reflection and self-evaluation 


CSC460 BS 


CSC460 BS 


CSC460 


Name: 


Exercise B5 (Module B Comprehensive Exercise), 2/25/05 


Note: This exercise is announced in advance so that we can think about it through this module. 
The due date of this exercise is the date of Module B Evaluation Workshop. 


Part 1: Analyzing Computability 


One of the main goals of Modules A & B (and the Theory of Computability as a whole) is to be 
able to identify the computability class of real-world problems (so that we won’t write a general 
program for an undecidable problem). In this module, we have been analyzing the main 
problems such as the halting problem with respect to their universality, self-reference, 
countability, reduction, etc. The real test for us is whether we can develop and describe our 
thought process involved in such activities and apply it to a new problem. While this is a major 
problem in the Theory of Computation, it is unlikely that you will easily find an answer in the 
literature (what are computer scientists doing?). By all means, it is a challenge. So, you are not 
expected to come up with a rock solid answer, not to mention a correct answer. Do your best. 
As usual, you are encouraged to discuss with other students and the instructor (but your writing 
must be your own, reflecting your own thought, which must be unique to yourself). If we as a 
class come up with useful analytical tools, it would be great. 


Note: Since Tasks 1 & 2 are closely related, you should tackle them in parallel, or in a circular 
manner (back and forth). 


Task 1: Develop your own “heuristics” to analyze a given problem with respect to the 
computability classes, i.e., decidable, semi-decidable, or non-TM-recognizable. By “heuristics,” 
we mean a speculative, but reasonable method of analyzing any given problem. For example, 
one might consider a rule such as: (i) if self-reference is involved, the problem is undecidable, or 
(ii) if the problem is uncountable, it is undecidable (these may or may not be true or even 
relevant; they are just examples of how to phrase a component of your method). 


Task 2: This task is an application of the previous one. (A) Identify some unique practical 
problem (in or outside computer science) which would be a good example of demonstrating your 
heuristics in Task 1. (B) Then, provide the set representation of the problem. (C) Finally, apply 
your heuristics developed in Task 1 to the problem to analyze its computability class. 


Part 2: Evaluation Form and Supporting Notes 


Review the evaluation procedure. Then, complete your evaluation form and supporting notes. 
Bring them to the evaluation workshop (hard copy). Print them well in advance so that you can 
avoid potential problems, e.g., not being able to print just before the evaluation. 


Survey: Time spent between after Unit B5 before the evaluation workshop: 


// End 
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Module B Review Exercise, 3/1/05 


(1) Do the required problems and (2) analyze and explain the computability class (e.g., 
decidability) of one of the optional problems, applying your own heuristics. In case your own 
problem in the comprehensive exercise is similar to any of the optional problems, you must try a 
different problem. Write your response on back or on another sheet of paper. Note: You are not 
supposed to solve any of the problems. 


Required Problems: ‘$’ Detection 


Analyze and compare the computability classes of the following two problems: 
A. To find out whether a given string contains at least one instance of the symbol ‘$’ 
B. To find out whether a given TM decides on Problem A (above) 


Option 1: Virtual Memory Paging (Architecture) 


To come up with an algorithm to find out whether page fault is avoidable, given a program, 
memory specification, and the specification of the applicable machine architecture 


Option 2: Deadlock Prevention (OS) 

To come up with an algorithm to prevent deadlocks, given a set of processes (which may spawn 
subprocesses) and resources (which may become unavailable) 

Option 3: Optimization (Compilers) 

To design a procedure to generate the most optimized target code of the given source code with 


respect to the target code size 


Option 4: Protocol Security (Networks) 


To develop a process to find out whether a given network protocol is secure 


Option 5: Semantic Errors (Databases) 


To identify a mechanism to detect semantic errors (e.g., contradictory queries, which would 
return an empty set) in database queries 


Option 6: Planning (AI) 


To develop a program to construct a plan (of actions) that would lead to the desired goal state, 
given a set of primitive actions 


Option 7: Best Solution 


To find the best solution, given a problem and some specification about the goodness of a 
solution 


Option 8: Decidability Classes 


To identify the computability class of a given problem 


// End 
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Exercise B6/C0, 3/1/05 
This exercise is due on Tue., Mar. 15, 2005 (..e., after the spring break). 


The Computational Power of Turing Machines 


We have been using the Turing machine as our model of computation. It is convenient because 
with TMs, we can capture all of (and exactly) the algorithms as we informally understand 
(Church-Turing thesis). We also discussed that nontrivial properties of TMs are undecidable, 
Le., not captured by the TM (Rice’s Theorem). In addition to the main problems, we noted more 
problems in this category in Module B Review Exercise. So, we know that TMs are limited, esp. 
when we need to deal with, say, practical questions involving “big” issues in the CS industry. 

On the other hand, we can easily imagine situations where even the TM seems to be overkill. 
For example, to check whether or not a given string is an acceptable variable name (e.g., in a 
compiler), we do not need the full power of the TM (1.e., no need for the infinite tape). If we can 
solve a problem with an easier means, we should use it instead of unnecessarily complicated 
means (“principle of parsimony” and “Ockham’s razor’). The ability to identify and use the 
minimal mechanism for a given problem is essential in virtually any area of study or work. Do 
you have this skill? This is another property that would distinguish computer scientists from 
programmers. “Elegant” solutions are often sought not because of esthetics but because of 
practicality, e.g., for easy understanding and maintenance. 


Task: Write a concise essay about whether or not the TM is an appropriate model with respect to 
its computational power, referring to (i) your problem from Exercise 00 and (ii) your mini 
research question(s) from Exercise A6/BO or different one(s) from the list (below). In addition 
to discussing these problems in their entirety, you must also consider possible subproblems (cf. 
Note 1 below). Apply the skills learned in earlier CS courses, e.g., object orientation and 
modular programming, and analyze whether any of the problems/subproblems would need the 
full power of the TM. 


Note 1: We know that compilation is a decidable problem. Thus, a TM is surely sufficient for 
the problem. But as we briefly discussed in class, a compiler typically consists of multiple 
phases/modules. When these phases are considered as subproblems, they would require 
mechanisms of different complexity. For example, the lexer requires a DFA (but not a full TM 
or even a stack), the parser requires a stack (but not a full TM), and semantic analysis (e.g., 
variable reference) requires some sort of table (may correspond toa TM). This way, when we 
deal with a simpler problem, e.g., lexical analysis, we can do it as easily/fast as possible. Ina 
similar manner, your problem(s) may be broken into subproblems, which may require different 
mechanisms of different complexity. 


Note 2: For each of the research questions listed below, there is some connection to the Theory 
of Computation. Although algorithmic computation is limited as we observed in Modules A & 
B, being able to identify such limitation could let us pinpoint the source of complexity beyond 
the traditional computation. With this ability, one will be able to learn a broad range of subjects 
in a systematic manner. When I first studied the Theory of Computation on my own almost two 
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decades ago (no CS courses taken and no plan to study CS at that point), the motivation was to 
understand the mechanism behind human language (i.e., linguistics). But the Theory certainly 
changed the way I understand human language (well, at least certain systematic aspects of it). 
When I started my graduate work in CS and studied CS topics such as programming languages, 
compilers, etc., I felt “OK.” But when I studied cognitive science and complex systems, it was 
the limitations of the Theory of Computation that I appreciated. It may be difficult to “use” the 
Theory in this manner, esp. during this semester (we do not really discuss much, outside the 
Theory topics in this course). But I hope you keep this in mind and try to apply the acquired 
ideas throughout your career. Whether you can connect apparently different things and analyze 
them systematically would make a huge difference in your life, regardless of your career choice. 


List of sample research questions from Exercise A6/B0 

1. Can organizational dynamics be modeled as an algorithm? 

2. Can evolution be modeled as an algorithm? 

3. Can ecology be modeled as an algorithm? 

4. Can human development be modeled as a computer? 

5. Can our minds be modeled as a computer? 

6. Can vision be modeled as an algorithm? 

7. Can learning be modeled as an algorithm? 

8. What would be the minimal mechanism to process human language? 

9. Can the entire situation of an arbitrary game be modeled as an algorithm? 

10. Would “perfect” user modeling, e.g., for web search, be possible? 

11. Would “perfect” computer security be possible? 

12. Can the entire process of software engineering be modeled computationally? 

13. Can computer networks be modeled as a single computer? 

14. Could biology be reduced to physics? 

15. Could some computer generate real (not pseudo) random numbers? 

16. Would it be possible to decide whether the given numbers are random? 

17. Would randomization affect computability and/or complexity? 

18. Would parallelism affect computability and/or complexity? 

19. Would artificial neural network be more powerful than TMs? 

20. Would cellular automata be more powerful than TMs? 

21. Would the use of analog (or fuzzy) values affect computability? 

22. Would relativistic, quantum, or some other modern-physics-based computation surpass TMs? 

23. Can all the cases of on-line algorithms be simulated by off-line computation? [On-line algorithms would 
obtain inputs as the time progress. Off-line computation would provide all the possibilities as input at once. Cf. 
function-to-relation conversion used to fold the output within the input] 

24. Would oracle computing affect computability? [Oracle computing: A TM with the capability to ask questions 
to another mechanism] 

25. Would persistent TM be able to compute more than the standard TM? [Persistent TM: Multiple sessions of 
TM operation with some memory between them] 

26. Would accelerating a TM give more power? 

27. Would slight error tolerance affect any aspect of the Theory of Computation? 

28. What would be the ability of a finite automaton with a queue? 

29. What would be the effect of “constant” (as in complexity analysis) in practice? 

30. Can any mathematical function be represented computationally? 

31. What exactly are power sets doing to the Theory of Computation? 

32. Is what you can do in logic the same as what you can do with computation? 

33. If you have a research question of your own, please consult the instructor first. 


// End 
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Unit B6 Supplement, 3/4/05 


Ex B5 Compiled Heuristics 


Here is a (edited and expanded) collection of ideas from your Ex B5, classified according to the 
involved techniques. In some cases, wording is subtle. 


TM Construction [Useful for simple problems, e.g., combinatory problems] 

1. If it is possible to create an algorithm (a terminating TM) to solve the problem, it is 
decidable. 

2. If itis possible to create a TM to “recognize” the problem/language (but not guaranteed to 
terminate), it is TM-recognizable. 

Note: The following cases are insufficient as conditions. 

e Not being able to come up with an algorithm/TM (in the future?) 

e Potential of looping (too informal; why did we discuss the main problems in detail?) 

e Solving at least one input (a single input is not enough) 


Set operation [Useful when multiple properties are known] 

If the problem is TM-recognizable and undecidable, it is semi-decidable. 

If the problem is both TM-recognizable and co-TM-recognizable, it is decidable. 

If the complement of the problem is decidable, it is decidable. 

If the complement of the problem is semi-decidable, it is non-TM-recognizable. 

If the complement of the problem is non-TM-recognizable, it is undecidable. [added] 


SON 


Reduction [Useful when another problem is known] 

8. Ifthe problem can be reduced to a known decidable (TM-recognizable) problem, the former 
problem is decidable (TM-recognizable). 

9. Ifa known undecidable (non-TM-recognizable) problem can be reduced to the problem, the 
latter problem is undecidable (non-TM-recognizable). 

10. If the computability analysis of the problem can be reduced to that of a known decidable 
(TM-recognizable) problem, the former problem is decidable (TM-recognizable). [added] 

11. If the computability analysis of a known undecidable (non-TM-recognizable) problem can be 
reduced to that of the problem, the latter problem is undecidable (non-TM-recognizable). 
[added] 


Countability [Useful when there are more languages (uncountable) than all of TMs (countable)] 

12. If the problem is about a non-trivial property of a TM/language, it is undecidable (Rice’s 
theorem). 

13. If the problem is a non-empty, proper subset of an uncountable set (or the power set of a 
countably infinite set), it is undecidable. [added] 

Note: The following cases are insufficient as conditions. 

e Uncountable set (there are trivial decidable problems, i.e., constant problems) 

e Countably infinite set (cf. Ly is countable) 

e Finite set (cf. a truly random problem) 


Other [Useful for complex representations, e.g., algorithm, function, logic, grammar] 
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14. If the problem involves a mechanism that allows counting (natural numbers) and self- 
reference, it is (probably) undecidable. [I am not prepared to do a rigorous justification. ] 

Note: The following cases are insufficient as conditions. 

e Universality or self-reference (this could happen in a limited domain which cannot count) 


// End 
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Module B Extra Problems, 3/4/05 


This is an optional problem set. These are provided for those who want to try some textbook- 
style problems. The problems are actually taken from a classic textbook. If you want to discuss 
these problems, contact the instructor. 


X1. Consider a model of a Turing machine in which each move permits the read-write head to 
travel more than one cell to the left or right, the distance and direction of travel being one of the 
arguments of 5. Give a precise definition of such an automata and sketch a simulation of it by a 
standard Turing machine. [Note: 6 is the transition function. ] 


X2. Show that the following problem is undecidable. Given any Turing machine M, a € T, and 
w € &", determine whether or not the symbol a is ever written when M is applied to w. [Note: 
“y”” is a regular expression, i.e., a sequence of one or more symbols in the set ¥.] 


X3. Show that the following problems are undecidable: 
A. L(M) contains any string of length five. 

B. L(M) is regular. 

[Note: L(M) denotes the language accepted by a TM M.] 


X4. Show that there is no algorithm for deciding if any two Turing machines M; and M) accept 
the same language. How would your conclusion be affected if M2 is a finite automaton? 


X5. Let B be the set of all Turing machines that halt when started with a blank tape. Show that 
this set is recursively enumerable, but not recursive. [Note: Review the terms.] 


X6. Let M; and M2 be arbitrary Turing machines. Show that the problem “L(M,) c L(M2)’ is 
undecidable. 


X7. Determine whether or not the following statement is true: Any problem whose domain is 
finite is decidable. 


X8 [Post’s Correspondence Problem (PCP)]. Consider a non-empty set of pairs of strings S = 
{(X1, V1)5 (X2, V2), «+» Xe, Ye) }. The problem is to find some sequence of members of S such that 
the concatenation of the x strings are identical to that of y strings. lv; = Ily; where ITI denotes the 
concatenation of strings and i denotes the index of a string. Show that PCP is undecidable. 
[Note: Do web search for more details about this well-known problem. ] 


// End 
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Module B Evaluation 


Review your portfolio 


+ My comments are sporadic and scattered 
on Review Ex, Comprehensive Ex, Supp. 
Notes, and Reflective Essay. 


« You are encouraged to clarify and discuss 
my comments. 


* You can keep the folder till the next class; 
then, return it to me. 
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Correction on Eval Form B 


The algorithmic notion of computation can be 
represented in a variety of equivalent forms, 
which define a bounded class of sets. That is, 
there is a limit to algorithmic computation. 
[computability] 


c. Can explain logically that the halting 
problem is semi-decidable and infinite-loop 
detection is uasetvabte non-TM- 
recognizable. 
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Module B Comprehensive/Review Exercise 


Required Problems: ‘$’ Detection 
A. To find out whether a given string contains at least one 
instance of the symbol ‘$’ 
B. To find out whether a given TM decides on Problem A 
Option 1: Virtual Memory Paging (Architecture) 
* Option 2: Deadlock Prevention (OS) 
Option 3: Optimization (Compilers) 
Option 4: Protocol Security (Networks) 
* Option 5: Semantic Errors (Databases) 
Option 6: Planning (Al) 
Option 7: Best Solution 
* Option 8: Decidability Classes 
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Unit B7: Overview 


« Review 
— Comprehensive/Review Exercises, etc. 


¢ Explore the notion of “computation” via the 
Church-Turing thesis 
— Discuss equivalence of different mechanisms 
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Review 


Computability Classes 


¢ With respect to impossibility, problems can 
be classified into 3 disjoint sets, based on 
whether a TM exists and/or always gives 

an answer. 

Undecidable gleich 


Semi-decidable 


TM-recognizable 
Decidable 
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Church-Turing Thesis 


* Hypothesis as definition 
Computation/effective procedure [informal] 
= TM operation [formal] 


¢ A variety of models of computation are 
equivalent to TM. 

* All sorts of (algorithmic) computation are 
created equal [and do the same thing, 
computation]. 
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Take-home exercises 


Connections 


¢ TMs (including variants) 
* Computers 
— Different programming paradigms 
¢ Unlimited Register Machine (URM) 
« Recursive functions 
¢ i-calculus [behind LISP] 
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Proof of Equivalence 


Equivalence of models M and N 
—Nsimulates M and M simulates N 

Set identity, e.g., A=B 

-AcBandBcaA 

Equivalence of propositions, e.g., p< q 
-p>q q>pP 

Equality of values, e.g., x= y 


—-xsy ysx 
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Instructions: 
— Zero: Z(i) > r, 0 
— Successor: S(i) > 7, 7; + 1 
— Transfer: TQ, j) => Ra" 
— Jump: Ji, j,n) => 
if r;=17;; proceed to the nth instruction 
otherwise: proceed to the next instruction 
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Preview 


Equivalence Summary 


Computers, 
i) 


v 


TM variations >TM > A-calculus 


URM «———> Recursive Functions 
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TM Variations 


Both accept and reject states (as well as 
other dying states) 


Semi-infinite tape 

Multiple tapes 
Nondeterministic transition 
—le., 6as a relation, not function 
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URM (2) 


« Program: A finite sequence of instructions. 
Sequential execution of instructions except 
for possible jumps 

* Initial values: Specific values can be 
assumed in the registers. 

* Convention: The output in R,. 
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Recursive Functions (1) 


¢ Built up from the basic functions by a finite 
number of operations of substitution, 
recursion, and minimalization: 
Basic functions 
— Zero: 0 
— Successor: x +1 
— Projection: U", (x, X, ..., %,) =%; 
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TM as Function 


A TM can simulate a function. When accepts 
¢ Function on strings 
— Input: A list of strings on the tape 
— Output: A string on the tape 
Function on natural numbers 
— Input: A list of natural numbers on the tape 
— Output: A natural number on the tape 
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Equivalence Summary 


Computers, 
“ 


v 
TM variations > TM > A-calculus 


URM «———> Recursive Functions 
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Recursive Functions (2) 


* Operations 


— Substitution (composition): Given computable 
functions fvy1, ..., yk) and g,(x), ..., g,(x), define 
h(x) = f(g, (2), --5 8,) 


— Recursion: Given computable functions f(x) 
and g(x, y, z), define 


* h(x, 0) =f(x) 
* h(x, y + 1) = gy, AQ, y)) 
— Minimalization: Given computable function f(x, 
y), define h(x) = the least y such that f(x, y) = 0 
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A-calculus 


Formal representation of functions 
¢ Abstraction: Ax.exp 

* Application: exp, exp, 

* B-reduction: (Ax.exp) y 


— The result is exp where the instances of x is 
replaced with y. 


—E.g., (Axfif))) 2= Af) is 2a function? 
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TM vs. Computers 
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Philosophy of Mind 


¢ Hypothesis: Mind is a computer [Fodor: 
specifically uses TMs as the model]. 


Unit Summary 


« Review 
¢ Church-Turing thesis 


— Defines the informal notion of computation 


— Leads to the equivalence of various 
algorithmic computational approaches 


CSC460 B7 CSC460 B7 


Computability Summary 


Main goal: To be able to analyze the 
computability class and identify what we can do 
with algorithms [Comprehensive Ex] 

Topics 

— Problem as a set 

— Turing machines and (un)decidability 

— Mathematical tools: diagonalization, power set 


— Halting and other main problems; Rice’s theorem 
— Reduction 


— Church-Turing thesis 
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Unit B7 Supplement, 3/7/05 


Part 1: Church-Turing Thesis 


As we discussed in class, the Church-Turing thesis states that the informal notion of computation 
is captured by Turing machines. Although it is called a “thesis” (i.e., proposition), it involves an 
informal notion, which can never be “proven.” That is why, it should be considered more as a 
definition. However, due to the equivalence between Turing machines with a wide range of 
other formalisms, most TM variants, real computers (with qualifications), Unlimited Register 
Machines, recursive functions, lambda calculus, etc., this thesis indeed captures the essence of at 
least algorithmic part of computation (more about non-algorithmic computation later). That is, if 
there are so many different ways to represent the same notion, albeit informal, such a notion 
must have something deep. For example, through the connection between TMs and recursive 
functions, we can often translate results in computer science and mathematics back and forth, 
providing different perspectives useful in different contexts. 


Here are examples of some other abstract models (see the definitions in Exercise B4 and Unit 
B5/B7). If you skim the examples below, you may realize why the TM is the most popular 
model of computation. 


Unlimited Register Machine (URM) 
e Example: Addition 7; + 72 
o Reset r3 = 0, with Zero 
o Count up 7; and 73 until 73 = 72 , iterating the basic operations including Successor and 
Jump 
o The result of the addition is in r;. 


The simulation of a URM with a TM is fairly intuitive. The other direction requires a few tricks. 
For example, we will need to be able to handle all symbolic computation with URMs; this can be 
done by associating all the tape symbols with unique natural numbers (ASCII code?). A URM 
can then represent the TM tape within its register, encode the tape position in a special register, 
simulate the basic TM operations as subroutines, etc. 


Recursive Functions 
e Example: Addition add(x, y) 
o add(x,0)=x 
o add(x, y + 1) = Successor(add(x, y)) [using the basic function Successor] 
e Example: Multiplication mult(x, y) 
o mult(x, 0) =0 
Oo mult(x, y + 1) = add(mult(x, y), x) [using pre-defined functions] 
e Example: Factorial fac(x) 
o fac(0)=1 
o fac(y + 1) = mult(fac(y), Successor(y)) [using pre-defined functions] 


The simulation of a recursive function with a TM is fairly intuitive. The other direction can go 
indirectly through the simulation of a URM with a recursive function. This simulation is still 


1/3 


CSC460 


involved. First, it is not very intuitive to simulate certain URM operations with a recursive 
function (e.g., Jump as a recursive function?). One way to cope with this situation is to limit the 
simulation to the cases where the given URMs that can compute a function (if not, we can 
implementing a looping function). So, by this assumption, we know that the URM (program) P 
receives inputs X = Xj, X2, ..., X, (boldface for a list or “vector’’) in the respective registers and 
computes the result P(x) to be left in Rj. Let us define two functions. First, result(x, t) =r; 
computing the result. Second, next(x, t) would return 0 if the computation terminates, or would 
return the next step (e.g., a function ID) if the computation still continues. If a function is 
defined, P(x) must terminate at some step 1; (1.e., ¢; = the least t such that next(x, t) = 0), and the 
result result(x, t;) appears Rj. Otherwise, P(x) would loop (i.e., next(x, t) #0 for any f). In either 
case, the desired function (which is computed by P) is defined as a recursive function result(x, 
the least ¢ such that next(x, t) = 0) 


Lambda Calculus 
e Example: Identity function 
Oo Axx 
e Example: Church numeral 
o Representation of a natural number n: Af-Ax.f"(x) 
O=AfiaAx.x [Note: AfAxf°(x)] 
Succ = An.Af.Ax.(f (1 f) x)) 
1 = Succ 0=AfAx.(f ((Af-Ax.x) f) x) = Af Ax.(f (Axx x)) = Af rx.(fx) 
[Note: Af.Axf |(x)] 
e Example: Conditional 
o T=AvxAy.x [i.e., picking the first argument] 
o F=AxAy.y [i.e., picking the second argument] 
o BMN (df B then M else N) where B is T or F 


oO O 


The simulation of lambda calculus with TMs would be intuitive, although tedious as usual. The 
other direction can go through recursive functions. 


Part 2: Rice’s Theorem 


Although you may not have noticed, many of the problems in Module B Review Exercise 
(Required Problem B, Options 1 through 6) can be answered very concisely by using Rice’s 
theorem. As the option problems indicate, many general problems in computer science refers to 
a non-trivial property of a certain general procedure. Thus, this theorem is widely applicable and 
useful to pinpoint the limitations of algorithmic computation. This is a review of this theorem 
with some insight into why this theorem holds. First, the definition again. 


Rice’s Theorem: A non-trivial property of a TM is undecidable. 


Note that a “non-trivial” property refers to any property of a TM that is not “constant” in the 
following sense. If the problem is to return “‘yes” or implement a constant function (e.g., Zero), 
the behavior is fixed and always decidable. Such a constant behavior is considered “trivial.” 
Everything else, including very simple problem such as telling if the input has at least one ‘$’ 
(Comprehensive Exercise Required Problem A), is non-trivial. We also know that each TM is 
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associated with a language which the TM at least “recognizes” (but not necessarily decides). 
The operation of a TM can also be associated with an algorithm (if terminates), procedure, etc. 
So, we might consider the following corollary as a slightly more applicable form of Rice’s 
theorem. 


Corollary: A non-constant property of a TM, language, algorithm, procedure, recursive function, 
etc. is undecidable. 


Let us see why this is the case, using Comprehensive Exercise Required Problem B, defined as 
TMpotar = {M| TM M accepts a string that contains at least one ‘$’}. First this is a problem 
about TMs (cf. Required Problem A, which is a problem about strings). Although it is not 
explicitly indicated as in ACCEPT ™ = {(M, w) |TM M accepts w}, TMpozzar Still needs to pair 
the given TM with all sorts of string inputs. But the sequence of strings given to the TM, 
interpreted as a language, may be a non-TM-recognizable one. Then, regardless of the TM at 
work at that point, such an input cannot be recognized. So, this problem must be undecidable. 
In a similar fashion, we can intuitively understand that Rice’s theorem and the corollary above, 
referring to any nontrivial property of a TM. 


For this particular problem TMpozzar, We can narrow down the computability class further. 
Since it would be possible to create a modified UTM that simulates the given TM on an input. If 


the modified UTM evaluates that the given TM detects at least one ‘$’, we can pick up all the 
positive cases. Thus, it is semi-decidable, as in ACCEPT yy and HALT 7 ™. 


// End 
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Module B Evaluation Unit C1: Overview 


Review your portfolio Overview Module C 

+ My comments are sporadic and scattered Discuss Ex B6/CO “The Power of TMs” 
on Review Ex, Comprehensive Ex, Supp. Introduce another model of computation 
Notes, and Reflective Essay. (equivalent to the TM) 

* You are encouraged to clarify and discuss Explore the effects of downgrading TMs 


be Identifying the minimal mechanism for the 
* You can keep the folder till the next class; given problem 


then, return it to me. Preview Exercise C1 “Chomsky Hierarchy” 


cscaso 1 csc4s0 ct ‘rough intro; more details later 


Overview: Theory of Computation Module C Goals (1) 


impossibility |Computability Content Goals 
5. Formal languages and automata 
Uninteractable? a. “Hierarchy” of TM-recognizable languages 
. Unrestricted grammars (rewriting system) ~ TMs 
ie eee .  Context-free languages (CFLs) ~ CFGs ~PDAs). When/how 


i i ei e i to use this class to analyze problems. 
cidabl F ocus In Module Cc 
Dist ibutivity? Undeci ca = - ' ts 


Regular languages (regular sets~ RegExps~ FSAs). 
When/how to use this class to analyze problems. 


3 . _ Show that some language is not regular. 
Complexity Tractable a . Show that some language is not context-free. That certain 


real practicality i? 
Intractable ontext-free 
- Say 


x roperties of CFLs are undecidable. 
icalii Languages} cin: P 
practicality gree simplicity Could explain this goal to CS students outside this class. 
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Module C Goals (2) Ex B6/CO: The Power of TMs 


Content Goals The entire problem 
7. Power set j j 
a. Nondeterminism as the power set of the possible states. * TMs are the appropriate mechanism. 
b. Nondeterminism with respect to TMs, PDAs, FSAs. . TMs are too weak. 
Performance Goals 
. rful. 
4. Critical attitude PM Satotce Peale 


a. Critically analyzed the usefulness of the “languages/automata” Modules/subproblems 
area of the (traditional) Theory of Computation 
5. Communication 
a. Completed all the exercises on time (take-home and in-class). 


b. Made conscious efforts to promote transfer of learning among 
students 


CSC460 C1 CSC460 C1 


Power-Economy Tradeoff 


* Complex mechanisms are more powerful 
than simple ones (can represent more 
variety of problems/languages). 

* Simple mechanisms are less expensive 
than complex ones (can operate simply 
and faster in general). 

« An appropriate level would be the least 
powerful one for the requirement. 


CSC460 C1 


Review 


TM-Equivalent Models 


Computers, 
‘ 


Y 
TM variations TM > A-calculus 


Recursive Functions 


csca46o C1 Other models? 


Grammar 


¢ Grammar [unrestricted grammar, semi-Thue system] 
— Rewrite rules of the form: a — B 
where a, B are strings of symbols and |a| > 0 


*S—> NP, VNP, 

* NP, V NP, > NP, is Ved by NP, 
¢ NP, = liisa | tiina 

* NP, > mikko | seppo 

* V > kisses | kicks 

* Ved > kissed | kicked 


csc4e0 C1 TM vs. grammar? "1 


Downgrading TMs 


* Imposing a specific tape organization 

— As a stack 

— As a queue 
* Limiting the tape size 

— Set some finite fixed bound 

— Limit the growth (e.g., linearly with the input) 
* Limiting the head mechanism 

— Read-only 


— One-way scanning only 
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Identifying Alternative Models 


« Problem ~ set 
— Most commonly as a language (set of strings) 
— Recognized by a TM 

¢ Whatever mechanism that can recognize 
(or generate) a string can be examined for 
the equivalence with a TM (or some 
downgraded forms). 
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Using a Grammar 


* Generating a string 
* Accepting a string (parsing) 


*S NP, VNP, 

* NP, V NP, > NP, is Ved by NP, 
* NP, = liisa | tiina 

* NP, > mikko | seppo 

* V > kisses | kicks 

* Ved > kissed | kicked 


* Terminal symbols [lower case]: Cannot be rewritten 
+ Nonterminal symbols [UPPER CASE]: Must be rewritten 
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TM vs. Grammar 


* Equivalent! 
* l.e., both can recognize/generate exactly 
the same class of languages 
— TM-recognizable 
— Recursively enumerable (RE) [standard term in 
this context] 
* Proof idea 
—A grammar simulated by a TM 
—ATM simulated by a grammar 
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Grammar Variations 


* Conditions on the length of LHS (left-hand 
side) and/or RHS 


* Conditions on the type of string on the 


ae *S > NP, VNP, 


* NP, V NP, > NP, is Ved by NP, 
* NP, = liisa | tiina 

* NP, > mikko | seppo 

°V > kisses | kicks 

* Ved > kissed | kicked 
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Automata Hierarchy 


* Type 0: TM 
Type 1: Linear Bounded Automaton (LBA) 
— Tape space limited to the input size 
Type 2: Pushdown Automaton (PDA) 
— Tape as a stack (with a separate input tape) 
Type 3: Finite-State Automaton (FSA) 
— Head movement only to the right 


os0460o1 “schematics 5, 


TM or Grammar? 


Modeling computation 
Modeling function 


Classifying sets (including computational 
problems) 


Declarative specification 

Specification of programming languages 
Description of human languages 
Analysis of different classes 
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Grammar Hierarchy 


* Type 0: Unrestricted Grammar 

¢ Type 1: Context-Sensitive Grammar 
(CSG) 
¢ |LHS| <|RHS|_ [i-e., no shrinking] 

* Type 2: Context-Free Grammar (CFG) 
* |LHS| = 1 [i.e., ignore the context of rewriting] 

7 Type 3: Regular Grammar ~ Regular Expression 


— RHS has at most one nonterminal at the 
left/right edge. 
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Language Hierarchy 


* Type n languages are accepted/generated 
by Type n grammars and recognized by 
Type n automata. 

* Type 0: RE (TM-recognizable) 

¢ Type 1: Context-Sensitive Languages 
(CSLs) 

* Type 2: Context-Free Languages (CFLs) 

* Type 3: Regular Languages 


CSC460 C1 


Chomsky Hierarchy 
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Recursive Sets 


* Decidable = Recursive 

* Theorem: CSLs c Recursive Sets (i.e., 
proper inclusion) 

¢ Proof idea 
—CSLs c Recursive Sets 
— There is a recursive set that is not a CSL. 
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Power-Economy Tradeoff, Revisited 


* Higher-level grammars/automata are more 
powerful than lower-level ones (can 
represent more variety of languages). 

¢ Lower-level grammars/automata are less 
expensive than higher-level ones (can 
operate simply and faster in general). 

¢ An appropriate level would be the least 
powerful one for the requirement. 
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Group Exercise 1 


Identify the minimal type for the following 
* Pattern matching? 

* Programming language specification? 
« Representing human language? 
Discuss the following 

* Effects of nondeterminism? 


* Is the hierarchy exhaustive? 
— Where is the set of decidable languages? 


— Class like Type 1.5? 
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Extended Chomsky Hierarchy 


TM-recognizable (RE) Grammar/TM 
Decidable (Recursive) *<-----\~---- ?/Decider 
Context-Sensitive CSG/LBA 


Context-Free \-p-- AA CFG/PDA 
~- RegExp/FSA 
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Unit Summary 


Grammars 

Connection between grammars, automata, 
and languages (problems) 

Chomsky hierarchy 

Identifying the minimal mechanism for the 
given problem 
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Summary Question 


¢ What would be your guiding principle 
to identify the simplest 
mechanism/approach (not necessarily 
in CS)? Explain. 


* Questions/Comments/Suggestions 
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March 15, 2005: Alternative Class Activities 


In case the instructor cannot make the class this day, do the following activities together in class: 


1. Review your Module B Review Exercise and Exercise B5 heuristics, also using the compiled 
heuristic and B7 supplement on Rice’s theorem (available on-line). The students who were 
present on Friday, March 4 are invited to share the discussion on that day. 

2. Discuss the Church-Turing thesis, also using the B7 supplement (available on-line). Again, 
the students who were present on Friday, March 4 are invited to share the discussion on that 
day. 

3. Form groups of two (or three students). Discuss your Exercise B6/CO “The Power of TMs.” 
In particular, compare the entire problem and subproblems with respect to whether you 
would need the full power of the TM. You should also make reference to the content of the 
slide below. Then, compare the results altogether. 


Power-Economy Tradeoff 


* Complex mechanisms are more powerful 
than simple ones (can represent more 
variety of problems/languages). 


¢ Simple mechanisms are less expensive 
than complex ones (can operate simply 
and faster in general). 


* An appropriate level would be the least 
powerful one for the requirement. 


cSc460 C1 1 


4. Go over Module C Content and Performance Goals (available on-line; see the eval form in 
Unit C6). You do not need to print the document (hard copies will be distributed in the next 
meeting). 


5. Form groups of two (or three students). Discuss the effects of the limitations on TMs. Then, 
compare the results altogether. 


Downgrading TMs 


Imposing a specific tape organization 
—As a stack 

— As a queue 

Limiting the tape size 

— Set some finite fixed bound 

— Limit the growth (e.g., linearly with the input) 
Limiting the head mechanism 

— Read-only 


— One-way scanning only 
CSC460 C1 alternative models 8 


For the next class on Friday, March 28, 2005, write up a report of the in-class activities. 


// End 
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Name: 


Exercise C1, 3/15/05 


Part 1: Chomsky Hierarchy for Your Problems 


The Chomsky hierarchy can help you identify the minimal mechanisms (grammars/automata) 
that are appropriate for your decidable problem. For example, if you know that parenthesis 
matching can be specified by a CFG and processed by a PDA, you would use these instead of 
more powerful and thus more costly unrestricted grammar and TM. At the same time, you know 
that you cannot use a regular expression or a FSA to do the job. We will be studying the two 
classes “context-free” and “regular” more in detail in the coming units. In this exercise, you will 
revisit your own problems, where you may want to review what you learned in other courses 
(most notably “compilers”) and develop some intuition behind the hierarchy. 


Task: Revisit the problems (including the subproblems) you discussed in Exercise B6/CO. 
Wherever possible, classify the problems based on the extended Chomsky hierarchy (i.e., TM- 
recognizable, decidable, context-free, or regular). 


Part 2: Grammar and TM Variants 


As we discussed in Module B, all nontrivial properties of a TM are undecidable. That is, we 
cannot have a general mechanism to decide on them. On the other hand, simpler mechanisms are 
in general much more manageable. So, in many cases, it makes sense to consider and use 
simpler mechanisms where appropriate. In this part, you speculate the power of some 
downgraded versions of grammars and TMs. If you have questions, contact the instructor. 


Task: For each of the following TM/grammar variants, try to find out what class of languages 
(from the extended Chomsky hierarchy) can be accepted/recognized/generated. 


A. CFGs without empty productions (note that CFGs can have rules of the form A — €, where € 
is the empty string) 

B. CFGs with rules such that |RHS| < 2 (e.g., A > B C is acceptable, but A > B C D is not) 

C. TMs with 2 stacks along with an additional input tape (i.e., like PDAs except that they are 
equipped with 2 independent stacks) 

D. TMs with a queue along with an additional input tape (i.e., like PDAs except that the tape is 
used as a queue, not a stack) 


Hints 

e For the first two problems, try to transform an arbitrary CFG to the limited versions. If you 
can do that, the limited versions can still accept all CFLs. If not, they may be less powerful. 

e For Problem C, see whether you can access an arbitrary element in one stack using the 
second stack (if so, the 2 stacks can simulate a single infinite tape). For Problem D, see 
whether the queue can simulate a stack or an infinite tape. 

e You are not expected to use references. However, if you do, cite them. 


Survey: Time spent between classes: 
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Unit C1 selected slides for the in-class Group Exercise 1 


Grammar Hierarchy 


* Type 0: Unrestricted grammar 

* Type 1: Context-Sensitive Grammar 
(CSG) 
*¢ |LHS| <|RHS|_ [i-e., no shrinking] 

* Type 2: Context-Free Grammar (CFG) 
¢ |LHS| = 1 [i.e., ignore the context of rewriting] 

* Type 3: Regular Grammar ~ Regular Expression 


— RHS has at most one nonterminal at the 
left/right edge. 


CSC460 C1 14 


Automata Hierarchy 


* Type 0: TM 
* Type 1: Linear Bounded Automaton (LBA) 
— Tape space limited to the input size 
* Type 2: Pushdown Automaton (PDA) 
— Tape as a stack (with a separate input tape) 
* Type 3: Finite-State Automaton (FSA) 
— Head movement only to the right 
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Language Hierarchy 


* Type n languages are accepted/generated 
by Type n grammars and recognized by 
Type n automata. 


* Type 0: RE (TM-recognizable) 
* Type 1: Context-Sensitive Languages 
(CSLs) 


* Type 2: Context-Free Languages (CFLs) 
* Type 3: Regular Languages 
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Chomsky Hierarchy 


Type 0 TM-recognizable (RE) *<--------- Grammar/TM 
Type 1 --~: Context-Sensitive “<---------7----- CSG/LBA 
Typa2 \---/ Context-Free \fo4---- CFG/PDA 
Type 3 -S&~ Fnvnnnnenenns REGEXP/FSA 
ee Contributions of Chomsky 


// End 
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Exercise C1 


Part 1: Your Own Problems 
Part 2: Grammar and TM Variants 
A. CFGs without empty productions 
B. CFGs with rules such that |RHS| < 2 
C. TMs with 2 stacks 
D. TMs with a queue 
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Review 


Extended Chomsky Hierarchy 


TM-recognizable (RE) *<-------- Grammar/TM 
Decidable (Recursive}*<---\~-- ?/Decider 


CSG/LBA 


foo CFGIPDA 
Conennenes ROGEXP/FSA 
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Group Exercise 1 


* How would you implement the module to 
analyze the input expressions? 
— How to specify? 
— How to process? 
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Unit C2: Overview 


Analyze a realistic example involving a 
mini language 


Understand CFGs/CFLs 


— Grammars, languages 

Understand how to process CFLs 
Understand the effect of determinism 
Preview Exercise C2 “Context-Free” 
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Graphing Tool 


* Basic functions 
— Constant: e.g., 1 (integers) and 2.34 (.5 must be written as 0.5) 
— Factorial: only the fixed form of n! 
— Logarithm: log OptBase NonConstantExp (OptBase: const) 


— Power: Exp*Exp where Exp is a valid expression. Exp other 
than a constant or n requires parentheses. 


* Complex expression 
— Complex expressions can be formed by using the operators *, /, 


+, -, and parentheses ( and ). * and / takes precedence over + 
and -. The operators are left associative. 


* Notes 


— Spaces are ignored internally. 
— sqrt(n) must be entered as n40.5. 


CSC460 C2 http: //www.tcnj.edu/~komagata/Graphing 


Context-Free Grammar (CFG) 


« G=(N,T,R, S) 
— N: finite set of nonterminals [upper case] 
—T: finite set of terminals [lower case] 


— R: finite set of rules A > a where ais a string 
made up of the elements of N and T 


— S: start symbol <« N 
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Context-Free Language (CFL) 


* Derivation (generation) of a string ina 
CFG, G (from the start symbol S) 
— S +* w (i.e., zero or more rule application) 
— Also said: “G generates w” or “G accepts w” 


* L(G) = {w|S ->* w, a string made up of the 


elements of T} 
* CFL= {L(G)|Gis a CFG} 
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Shorthand 


+ Alternatives: A > a| 8 
“A>a 
“A+B 

* Optional element: A > o [B] 
“A>a 
*A>aBp 
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Bottom-up Parsing 


¢ Example: Shift-reduce parsing 
— If the RHS of a rule matches a part of the 
input, reduce it to the LHS symbol 


— Otherwise, push the leftmost symbol onto the 
stack and repeat (shift) 


csc46o c2 Essential component for processing? —,, 


Parsing 


* Parsing: Process of analyzing how a 


particular string can be generated by a 
grammar 


Top-down: Start from the “start symbol” 
Bottom-up: Start from the string 
Hybrid: Combination of both 
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Top-down Parsing 


Example: Recursive-descent parsing 


— Keep the current input and the remaining part 
of the rule on stack 


— Expand nonterminals 
— Check terminals against the input 


ON => Ay 
*A>aAblab 
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Main CF Property 


¢ Matching growth to the left and right 


-E.g., (...0)...), O"'1", ay... 


nA ys) 
Sa Sa 
n n 


Characterized by balanced rules 
-E.g.,A>(A),A30A1,A34,Aq, 
Characterized by the use of stack 


—E.g., pushing ‘( and later popping it and 
matching with ‘)’ 
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Push-Down Automata (PDA) 


PDA, Formally 


* M=(Q,4,T, 6, qd, F) 
— Q: set of states 
— =: set of input symbols 
—T: set of stack symbols 


ijvie[n|B/G 


Nondeterministic 
Machine with a finite number of states 
Some mechanism to control — 8: transition function Note: € as the empty string 
State 0 | State 1 | State 2 State k Ox (ZU {e}) xP > P(OXxT*) “power set 
| T*: string of stack symbols 

— qo: initial state € O 
‘Stack 


— F: set of final states c O 
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Equivalence of CFGs and PDAs 


¢ Simulate a CFG with a PDA 


« Are there any potential parsing problem 
* Simulate a PDA with a CFG the following grammars? 


A B Cc 

"SrA "STA "SrA 
*AraAblab *A>aAa *A>adAa 
“AE *A>b *“A>a 


Group Exercise 2 
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Ambiguity Deterministic PDA (DPDA) 


PDA where the range of the transition 


function is a set of singletons (unique 
state). 


— Equivalently, 
PDA: @x (Zu {e}) xT > P.O xT) 
DPDA: Qx (Zu {e})x [3 Oxl 
* Sufficiently powerful to characterize 
programming language core 
* Efficient parsing algorithms are known. 
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¢ Multiple derivations of a single string 
¢ Acomplete analysis requires 
nondeterminism. 
* Problems with respect to: 
— Semantics 
— Efficiency 


Deterministic CFL (DCFL) 


¢ A subset of CFLs that can be processed 
by aDPDA 
¢ No easily identified class of grammars 


— Practical issue for specifying a programming 
language 
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Unit Summary 


¢ Many mini languages can be specified and 
processed by CFLs and PDAs. 

* CFG: A single nonterminal on the LHS 
(e.g., A > a) 

* CFL: Specified by CFGs 

* PDA: Process CFLs 

* DPDA/DCEFL: Deterministic subset of 
PDAs/CFLs ~ backbone of programming 
languages 


CSC460 


Extended Chomsky Hierarchy 


TM-recognizable (RE) Grammar/TM 


Decidable (Recursive)*\-~----\----_ /Decider 


Context-Sensitive CSG/LBA 
salieri CFG/PDA 


( Gooulary Z RegExp/FSA 
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Summary Question 


¢ Can you see the “context-free” 
property in some real-life 
phenomena? Explain. 


* Questions/Comments/Suggestions 
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Exercise C2, 3/18/05 


Part 1: Review ‘“‘Context-Free”’ 


The notion of “context-freeness” is extremely important in computer science, due to its tight 
connection with programming languages. If you understand this class, you understand the 
backbone of programming languages. So, here are some review questions. It is up to you how 
to use these. If you understand well by now, you do not need to write up your responses. But 
recall that passive and active knowledge are different. 


Review questions 

. What is the most important property of “context-freeness”? 

Why the constraint |LHS| = 1 leads to a CFG (cf. unrestricted grammars)? 
What kind of strings cannot be specified by CFGs? 

. What are the differences between top-down and bottom-up parsing? 

Why can PDAs capture all of CFLs? 

Why is the class of DCFLs smaller than that of CFLs? 


7M Ow > 


Part 2: Graphing Tool (Unit C2 Group Exercise 1) 


When we analyze the growth rates of two functions, it would be useful to visualize how fast 
those functions would grow. The demo program shown in class 
(http://www.tcnj.edu/~komagata/Graphing) just does this. In this exercise, continue the in-class 
exercise and give a specification of the language the graphing tool would accept as a CFL. By 
trying various expressions, you should be able to tell the range of acceptable forms. Although 
you are encouraged to do this exercise from scratch, if you need something to start with, you can 
use the following partial information. 


e Expression is a series of Terms connected with + or — 
o Exp— Term 
o Exp — Term + Exp 
o Exp — Term — Exp 
Term is a series of Factors connected with * or / 
Factor is Const, Factorial, Log, or Power 
Const 
Factorial > “n!” 
Log 
o Log — “logn” | “log” Const “n’” | “log(’” Exp “)” | “log” Constant “(“ Exp “)” 
o Note: “log” Const Const is illegal 
o Note: not including “log(’ Constant “)(’ Exp “)” 
e Power 
o Power — (“n” | “(’ Exp “)” | Const) “”” (‘n’” | “C’ Exp “)” | Const) 
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Part 3: Foxtrot 


CFGs are quite useful for various things. In my Discrete Math courses, I used examples such as 
robot navigation, DNA analysis, dance choreography (sort of), etc. Here is another example 
from ballroom dancing. A collection of steps of “Foxtrot” can be described by the following 
CFG (only the rewrite rules are shown; other information can be derived from them): 


Foxtrot — Basic 

Basic — Basic Basic 

Basic — featherFinish WeaveSeq Turn 

Basic — promenadePosition naturalWeave Turn 

WeaveSeq —-> € (empty string, i.e., no steps) 
WeaveSeq — weave WeaveSeq 

Turn — threeStep heelTurn 


Here, Foxtrot (start symbol), Basic, WeaveSeq, and Turn (beginning with an uppercase letter) 
are nonterminals, and the others are terminals. We will and you may abbreviate 
nonterminals/terminals using the first letter (case sensitive), e.g., W for WeaveSeq and w for 
weave. 


A. Would the following step sequence be acceptable according to the above description? If yes, 
show how such a sequence can be generated. If no, explain. 


fthfwthpnthfwwwth 


B. Discuss the advantages/disadvantages of using (a) top-down parsing and (b) bottom-up 
parsing for this grammar. Note that you must contrast top-down vs. bottom-up approaches in 
general and should not be limited to some particular top-down or bottom-up mechanisms 
(i.e., do not discuss recursive descent, LL(1), LR(k), etc.). 


Note: You must clearly state some contrastive feature(s). 


C. Would it be possible to generate exactly the same language (the step sequences generated by 
the above-mentioned CFG) using a regular grammar? Explain. In addition, argue whether 
this language is a context-free or regular [your argument should be very brief]. 


Note: A regular grammar is like CFGs except that the RHS can have only one nonterminal at 
the beginning or at the end. For example, a regular grammar can have rules such as “A > a 
b C’ and “A > A bc”, but not “A — aBc” or “A > aB C” (uppercase symbols are 
nonterminals). Regular grammars is equivalent to regular expressions and finite-state 
automata. 


Hint: Examine several example step sequences and figure out the pattern of generated strings. 
Is the pattern similar to that of parenthesis matching, which requires a CFG? 


// End 
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Ex C2 Part 1 “GF” 


What is the most important property of “context- 
freeness”? 


Why the constraint |LHS| = 1 leads to a CFG (cf. 
unrestricted grammars)? 


What kind of strings cannot be specified by CFGs? 
What are the differences between top-down and 
bottom-up parsing? 

Why can PDAs capture all of CFLs? 

Why is the class of DCFLs smaller than that of CFLs? 
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Ex C2 Part 2 Graphing Tool 


Expression is a series of Terms connected with + or - 
+ Exp — Term Exp > Term + Exp Exp — Term - Exp 
+ Note: Exp > € 


+ Term is a series of Factors connected with * or / 


+ Term — Factor Term — Factor * Term Term — Factor / Term 


+ Factor is Const, Factorial, Log, or Power 


+ Factor — Const | Factorial | Log | Power 


* Const 


+ Const — Digits Const — Digits *.” Digits Digits — Digit 
+ Digits Digit Digits Digit 0/1]... 9 


* Factorial > “n!" 
+ Log 


+ Log “logn” | “log” Const “n” | “log(” Exp *)" | “log” Constant “(" Exp ")" 
+ Note: “log” Const Const is illegal 
+ Note: not including “log(” Constant “)(" Exp “)” 


+ Power 


+ Power = (“n” | “(" Exp “)" | Const) [ “*" (“n” | “(" Exp “)" | Const) ] 
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Top-down Parsing 


Example: Recursive-descent parsing 


— Keep the current input and the remaining part 
of the rule on stack 


— Expand nonterminals 
— Check terminals against the input 


“SoA 
*A>aAb|lab 
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Review 
Languages/Grammars/Automata 


Language: Set of strings ~ problems 
Grammar: Convenient way of specifying 
languages 

—E.g., L(G) fora CFG G 

Automata: Abstract model of processing 
languages 

—E.g., L(G), where G is a CFG, needs a PDA 
Chomsky hierarchy: about languages 
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Ex C2 Part 3 Foxtrot Question C 


* Possible to represent the same language 
with a regular grammar? 


Foxtrot — Basic 

Basic — promenadePosition naturalWeave threeStep heelTurn Basic 
Basic — featherFinish WeaveNext 

WeaveNext -—> weave WeaveNext 

WeaveNext -— threeStep heelTurn [ Basic } 
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Bottom-up Parsing 


¢ Example: Shift-reduce parsing 


— If the RHS of a rule matches a part of the 
input, reduce it to the LHS symbol 


— Otherwise, push the leftmost symbol onto the 
stack and repeat (shift) 


http: //www.tcnj.edu/~komagata/ShiftReduce/ 
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Main CF Property Group Exercise 1 (CF) 


¢ Matching growth to the left and right Which of the following languages is 

= E.g., (..().--), OV, 4... Aye context-free? Why? 

ai a . O71" 2k3k 
* Characterized by balanced rules » ON2KZKT" 
. 02910)" 

-E.g.,A>(A), A90A1,A 34, Aa, 
* Characterized by the use of stack 

—E.g., pushing ‘( and later popping it and 


matching with 9’ priaaKoaaKn Oars 
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Unit C3: Overview Extended Chomsky Hierarchy 


« Review examples of regular languages 


— Review grammars/automata for regular TM-recognizable (RE) 


Grammar/TM 
languages 


- Understand the effects of nondeterminism Decidable (Recursive}<-—-"\-~_ Decider 
for different automata 


* Preview Exercise C3 “Regular” 


Context-Sensitive---F-- CSG/LBA 
Context-Free} fo CFG/PDA 
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Initial survey problem 


Vending Machine Password Screening 


* Imagine a vending machine that accepts Requirement 1 
nickels, dimes, and quarters and sells 
Vodka by a paper cup for 35 cents. Draw recognizable. 


— Your password must be non-TM- 


a diagram of a finite-state 
machine/automaton that would represent 
the state of the vending machine. Make 
additional assumptions if necessary. 


Requirement 2 

— Your password must be a palindrome. 

Requirement 3 

— Your password must be at least 6 characters 
long, contain at least one uppercase letter, 
and at least one symbol or a numeral. 
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Java Comments Binary Arithmetic 


* Do we need a TM? 
* Do we need a PDA? 


¢ End-of-line comment: // ... (until EOL) 
* Traditional comment: /* ... */ 
— Note: No nesting allowed 
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FA Components 


Finite-State Automata (FSA) 


* Sequence * A machine that consists of the three 


components shown on the previous slide 
* Alternative 


* Repetition (zero or more times) 


oe 
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Deterministic FA (DFA) 


* M=(Q,%, 5, q, F) 
— Q: set of states wey two possible moves 
— =: set of input symbols / 
— 6: transition function 
Qxx>Q 


a b 
OO) 
— qo: initial state « Q —~ 


— F: set of final states c QO 


Nondeterministic Example 


€ 
empty string 


CSC460 C3 


CSC460 C3 


Tracing Example 1 


Time 0 Time 1 Time 2 Time 3 
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Nondeterministic FA (NFA) 


* M=(Q,5, 6, q, F) 
— Q: set of states 
— : set of input symbols 
— 6: transition function 


Ox uel) > PO) | ekranslion elation 


— qo: initial state € O 
— F: set of final states < O 
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Effects of Nondeterminism 


TMs: No effects on power (i.e., the set of 
languages that can be recognized) 


PDAs: Affects the power (i.e., PDAs # 
DPDAs) 


FSAs: No effects on the power (i.e., the 
set of languages that can be recognized) 


Efficiency/speed: exponential slow down 


CSC460 C3 


Observation 


Due to nondeterminism and €-transitions, 
the result of a move can be a combination 
of possible next states. 

For a set of states 0, such a combination 
is a member of the power set of Q, i.e., 
P(Q). 

—E.g., P({1, 2}) = {D, (1}, {2}, (1, 2}} 
Naturally, any DFA is an NFA. 
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Nondeterminism 


* Possibility of multiple transitions from a 
state on a single input 

* Instead of a single state, considering a set 
of states 
* Sas the set of all states 
* P(S) as the power set of S 
« The next state < P(S) 

* Power set ~ nondeterminism 
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Regular Grammar 


« G=(N,T,R, S) 
— N: finite set of nonterminals [upper case] 
— T: finite set of terminals {lower case] 
— R: finite set of rules A > a B or A > B a where 
* ais a string made up of the elements of T 
*BeN 
— S: start symbol e N 
* Regular languages = {L(G) | G is a regular 
grammar} 


CSC460 C3 


Regular Expression (RegExp) 


¢ Practical alternative to regular grammar 
* Expression components (with sets x, Y) 


— Sequence: X Y Notation: X" = X...X 
Ses 


— Alternative: X | Y 


— Repetition (zero or more times): X* Kleene closure 


¢ Equivalence of RegExp and regular 
grammars 
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Main Regular Property 


* Language: Uncoordinated repetition 
—E.g., 01”, “a very very .... long noun phrase” 
¢ Grammar: Characterized by recursion at 
edge 
-E.g.,A7aA,A>Aa 
— Equivalently, Kleene closure, X* 
¢ Automaton: Characterized by finite states 
—E.g., looping represented by a cycle 
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Unit Summary 


¢ Main property of regular languages: 
uncoordinated repetition, cf. CF 

* RegExp is a practical way of specifying a 
broad range of simple languages. 


¢ FAis a simple and fast model for 
processing regular languages. 
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Equivalence 


Regular Grammar 
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More Examples 


FP numbers 
URL 
Spanish spelling (cf. English spelling) 


Noun phrase in English (except for CF 
components) 


Musical code sequence (?) 
Your daily tasks 
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Summary Question 


¢ How far have you been thinking 
Exercise C5 (Comprehensive 
Exercise)? Explain. If not yet, what 
is your current thought? 


* Questions/Comments/Suggestions 
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Name: 


Exercise C3, 3/22/05 


Part 1: Language Identification 


Regular expressions are one of the most commonly used way to specify regular languages. Since 
most modern computing environment support the use of regular expression, it would be good to 
practice specifying a regular language using a regular expression. 


As an example, let us observe a very crude way to identify Japanese names from a list of names 
(http://www.tcnj.edu/~komagata/csc460/05s/misc/names.txt). The first approximation is to pick 
up the basic Japanese syllabic structure, which is a repetition of consonant-vowel sequence. A 
regular expression (CV) captures the pattern, where V = {a, e, i, o, u} and C= 
LowerCaseLetters — V. Using the Unix egrep utility, we can see how well this pattern picks up 
Japanese names. 


% egrep -ni '*([bcdfghjklmnpqrstvwxyz] [aeiou])+$' 
~komagata/www/csc460/05s/misc/names.txt 
83: Duda 
142:Ho 
150:Hu 
166: Kane 
177:Ko 
197: Levi 
198: Levine 
199:Li 
243:Pi 
245:Pulito 
250:Rice 
259:Sami 
299:Su 
324:Vita 
325:Vu 
345:Wu 
346:Xu 
348:Ye 
350:Yi 
351:Yi 
354:Yu 


Since most names are not really Japanese (although you may not be able to tell), this pattern is 
not really good. But the accuracy can be improved by fine tuning the pattern. 


Task: Pick one human language and create a single regular expression that would pick up the 
names in that language to some extent. Include the regular expression and a test run on the name 
list. Human languages are full of exceptions. So, limit your time and satisfy yourself at some 
point. 


Note: If you prefer, you can use an alternative programming language/environment that supports 
regular expressions, e.g., perl, PHP. 
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Part 2: Distinguishing Regular and Non-Regular Languages 


FSAs are extremely simple and fast automata. So, if your problem (set, language) is regular, you 
should definitely use an FSA, rather than a PDA or a TM as the abstract model for developing a 
program. Then, the question would be to analyze whether the problem is regular. 


Task 1: Consider the following problem: Liisa realized that for any day, she receives m 
important (1.e., non-junk) e-mail messages followed by n junk messages, where m + n = 2k for 
some integer k => 1. First, represent the problem as a set. Then, identify whether the problem, 
interpreted as a language, is regular or non-regular. Justify your answer at least to the level 
discussed in class. 


Task 2: Consider the following problem: Mikko realized that whenever he wins the card game m 
consecutive times, he loses n consecutive times after that, where n = 2m + 1. First, represent the 
problem as a set. Then, identify whether the problem, interpreted as a language, is regular or 
non-regular. Justify your answer at least to the level discussed in class. 


// End 
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Ex C3 Part 2 Unit C4: Overview 


* Task 1: Liisa realized that for any day, she Distinguish regular and non-regular 
receives m important (i.e., non-junk) e-mail languages 
messages followed by n junk messages, — Pumping Lemma (for regular languages) 
where m + n = 2k for some integer k > 1. Understand and use the properties of 
* Task 2: Mikko realized that whenever he regular languages 
wins the card game m consecutive times, 
he loses n consecutive times after that, 
where n= 2m + 1. 


— Closure properties 
— Equivalence of regular languages 
Preview Exercise C4 “Regular Practice” 
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Regular? Analysis of “Regular’ness 


Top-down approach to paper writing 


* To Show that L is “regular” 
Finite language 


— Proof by existence: Give a regular grammar, 
orn RegExp, or FSA 


o”1" | m,n > 0} * To Show that L is not “regular” 
oO" 1" | m#n} — Need to prove that no regular grammar (or 


ww | w® is the reverse of we {0, 1}*} — Demonstrate that some property of regular 
ww |we {0, 1}*} languages cannot hold 
{www | w® is the reverse of w € {0, 1}*} 
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{ 

{ 

{w$w* | w® is the reverse of w € {0, 1}*} RegExp or FA) can generate the language 
{ 

{ 
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Main Property of Regular Lang uages Pu mpi ng Lem Mai (for regular languages) 


¢ Uncoordinated repetition 


* Inputs longer than the number of FA states For any infinite regular language L 
= Some state(s) must be repeated : 


, there exists a positive integer n, such that 
« Any number of that repetition must result joanne teach anaihals 
in an acceptable input. reek ane [| 2 Mo» 
there exists a decomposition w = xyz where 
‘ct pumping gas for free lxy| <np and |p| > 1 such that 
for any i= 0, 
exyze L 
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To show that a language is nof regular 
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Game-Theoretic Interpretation of FOL 


e VWxdy (x kicks y) cf. logic- 
— Falsifier (tries to crack V): Choose x 
— Verifier (tries to support 3): Choose y, based on x 
— Check whether “x kicks y” is true 
e AxVy (x kicks y) 
— Verifier: Choose x 
— Falsifier: Choose y, based on x 
— Check whether “x kicks y” is true 


To satisfy: Play a verifier 
To reject: Play a falsifier 
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Revisited 


Main Property of Regular Languages 


¢ Uncoordinated repetition 


* Inputs longer than the number of FA states 
= Some state(s) must be repeated 


« Any number of that repetition must result 
in an acceptable input. 
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Misusing Pumping Lemma 


* To show that 01” has “regular’ness 
Choose ny = 1 
Anticipate any 01" « L such that m+n2>1 
Choose a decomposition w = xyz: 
— Case 1 (m=0):1" Ie 1¢4(c21) 
— Case 2 (m #0): 0¢ 0° 1" (b21) 
Anticipate any i = 0 
O01" € L, O14 € L 
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Group Exercise 1 


Satisfy/reject the following 
. Everyone in this class is taking at least 
one upper-level CS course. 
. Some cat loves every dog at some point. 
. Some cat loves every dog at any point. 


Procedure: 

* Take the role of either the verifier or falsifier 
* To satisfy, the verifier must win. 

+ To reject, the falsifier must win. 
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Logic Game for Pumping Lemma 


Between a verifier (for 4) and a falsifier (for V) 
¢ Falsifier: Choose an infinite language L 
¢ Verifier: Choose a positive integer n, such that 
+ Falsifier: Choose w e€ L such that |w| 2 no, 
¢ Verifier: Choose a decomposition w = xyz where 
kxy| <n and ly > 1 such that “Meaning of fey] <ny. 
Falsifier: Choose i= 0, 
Check whether xy'z € L 
To satisfy: Play a verifier (wrong) 


To reject: Play a falsifier (correct) 
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Using Pumping Lemma 


Play a falsifier: This is how this lemma is used. 
To show that 0"1” is not regular 
Anticipate any n, 2 1 
Choose 0"1" € L such that 2n = ny 
Anticipate any decomposition w = xyz: 
— Case 1: 0¢ 0) 1" (b2 1) 
— Case 2:0" 1¢ 14(c21) 
— Case 3:0¢ (01°) 14(b+c2 1) 
* Choose i=2 
© 09071" ¢ L, 0712-14 ¢ L, and 04(0'1°)71¢4 ¢ L 
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Regular? 


Top-down approach to paper writing 
Finite language 

orl" 

{0"1" | m,n = 0} 

{0"1" | m#n} 

{w$w* | w® is the reverse of w € {0, 1}*} 
{ww* | w* is the reverse of we {0, 1}*} 
{ww | we {0, 1}*} 

{www | w® is the reverse of w © {0, 1}*} 
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Closure 


Closed under 


Concatenation 
Union 

Kleene closure 
Complement 
Intersection 
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Unit Summary 


Regular vs. non-regular 


— To show regular: Construct a regular grammar, 
RegExp, or a FSA 


— To show not regular: Use the Pumping Lemma 
Properties 

— Closed under most common operations 

— Unique minimization possible 


— Easy to manipulate regular languages and 
their specification/processing mechanisms 
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Group Exercise 2 


Satisfy/reject the following 
A. {w$w* | w® is the reverse of we {0, 1}*} 
B. {ww* | w® is the reverse of we {0, 1}*} 


Procedure: 

* Take the role of either the verifier or falsifier 
* To satisfy, the verifier must win. 

+ To reject, the falsifier must win. 
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Equivalence of Regular Languages 


* Equivalence of States 


— Equivalent behavior for all strings with respect to 
acceptance 


— To find out: compare every pair of states against all 
strings (systematically) 


Minimization of DFAs 

— Merge equivalent states = partition 

Equivalence of regular languages 

— Merge two DFAs and test equivalence of the two start 
states 
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Summary Question 


¢ The Pumping Lemma is tricky. You 


must have questions. What are they? 


¢ List other questions as well, if you 


have. 
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Unit C4 Supplement, 3/25/05 


Pumping Lemma (for regular languages) 


Since I was not able to explain this well in class, I will add a few notes. In addition, the slides 
are in an abbreviated form. So, I will try to give more details here. 


First, the Pumping Lemma says the following (informally). Given an infinite regular language, 
we can always find a (sufficiently long) string (w) that has a substring (y) contributing to 
(uncoordinated) repetition, and that the substring can be pumped zero or more times (y*). 
Although we don’t prove this lemma, we can intuitively tell this is true. 


As discussed in class, this lemma is used only for showing that certain languages are not regular. 

To do this, we take the role of a falsifier and face a competing verifier. 

e Asa falsifier, choose an infinite (supposedly) regular language L = 0"1" (shorthand for {0"1" | 
n= O}). If this was indeed regular, the falsifier could never win. 

e The opponent, a verifier, chooses a positive integer no, say, 4. This could be any other 
positive integer, though. 

© We choose w = 0°1* € L , which satisfies |0717| = 4 > no = 4. 

e The opponent can decompose w. The opponent only needs to choose the decomposition 
most damaging to us. However, since we don’t know what it is, we consider all the 
possibilities of decomposing w = xyz, where |xy| < no = 4 and |y| = 1 

0 Case 1 (y stays within 07) 
=" Subcase A: x =0, y=0,z=11 
= Subcase B: x= ¢, y= 00, z= 11 
0 Case 2 (y stays within 17) 
=" Subcase A: x = 00, y=1,z=1 
= Subcase B: x = 00, y=11,z=¢€ 
0 Case 3 (y crosses 0717) 
=" Subcase A: x =0,y=01,z=1 
= Subcase B: x=, y=001,z=6€ 
=" Subcase C: x=0, y=011,z=¢€ 
= Subcase D: x = s, y=0011,z=¢ 
oO Note: Depending on the choice made in the previous steps, there may be more 
cases/subcases. But the idea is analogous. 

e We choose i=22>0. Then, for Cases 1 and 2, the balance between 0’s and 1’s will be 
destroyed. For Case 3, the 0-to-1 ordering will be destroyed. In any case, we defeated the 
opponent. This means that we have rejected the Pumping Lemma. However, since the 
lemma holds for any infinite regular languages, we conclude that the language is not regular. 
[Strictly speaking, this is a proof by contradiction. ] 


The problem w$w* can be analyzed in an analogous way. In particular, you might consider 
different cases: whether or not y includes $. 
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For ww*, a similar technique may not work well. Recall the class discussion, where you are 
forced to pump the middle section of, say, 001100. You can pump and will still get strings that 
are in the language. To see how to deal with this problem, you may want to re-visit 0"1" and re- 
examine the choices made by the players. There are ways for the falsifier to defeat the verifier 
more easily. For example, what would happen if you consider w twice as long as no (instead of 
|w| = no as in the earlier example)? If it leads to a simpler strategy, what makes it happen? Such 
an idea may be useful for showing that ww’ is not regular. 


If you have questions, try to post on the discussion board, although you can always send them to 


me. 


// End 
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Name: 


Exercise C4, 3/25/05 


Part 1: Review: First-Order Logic (FOL) 


The Pumping Lemma involves a complicated interaction of first-order quantifiers (V and 4). In 
order to understand how to use it, we need to know the basics of FOL. 


Task (optional): Review FOL, if necessary. 


Part 2: Game-Theoretic Interpretation of FOL 


One way to interpret complex first-order statements is to use the game-theoretic interpretation. 
Task 1: Write up Unit C4 Group Exercise 1. 


Task 2 (optional): “O’ (asymptotic upper bound) is defined as follows: f(n) € O(g(n)) if there are 
constants c > 0 and no 2 1 such that f(n) < c g(n) for every integer n = no. Use the game-theoretic 
interpretation to analyze this. 


Task 3 (optional): The limit c of a function f(x) as x approaches a, lim f (x)=c, is defined as 


follows: For every real number € > 0, there exists a real number 6 > 0 such that |f(x) — c] < € 
whenever 0 < |x —a| < 8. Use the game-theoretic interpretation to analyze this. 


Part 3: Distinguishing Regular vs. Non-Regular Languages 


Using the Pumping Lemma, we can prove that a certain language is not regular in a precise 
manner. For such languages, we will need to use a more powerful mechanism, e.g., PDA, TM. 


Task 1: Write up Unit C4 Group Exercise 2. 
Task 2: Redo the two problems (tasks) in Exercise C3 Part 2 using the tools discussed in Unit 


C4. Note that you will need to use different approaches to show that a language is regular or 
non-regular. 


Survey: Time spent between classes: 
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Unit C4 slides for the in-class group exercises and take-home exercies. 


Game-Theoretic Interpretation of FOL 


+ Vay (x kicks y) of. logic-structure connection. 
— Falsifier (tries to crack V): Choose x 
— Verifier (tries to support 3): Choose y, based on x 
— Check whether “x kicks y” is true 
© AxVy (x kicks y) 
— Verifier: Choose x 
— Falsifier: Choose y, based on x 
— Check whether “x kicks y” is true 


To satisfy: Play a verifier 
To reject: Play a falsifier 
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Pu m p i ng Lem ma (for regular languages) 


To show that a language is not regular 


* For any infinite regular language L, 

* there exists a positive integer n, such that 

* for any we L such that |w| > 1, 

* there exists a decomposition w = xyz where 
|xy| < my and |y| = 1 such that 

* for any i> 0, 

exyzeL 
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Logic Game for Pumping Lemma 


Between a verifier (for 4) and a falsifier (for V) 
* Falsifier: Choose an infinite language L 
* Verifier: Choose a positive integer ny) such that 
+ Falsifier: Choose we L such that |w| = no, 
¢ Verifier: Choose a decomposition w = xyz where 
|xy| $79 and |y| = 1 such that ree El 
¢ Falsifier: Choose i> 0, 
* Check whether xy'ze L 
To satisfy: Play a verifier (wrong) 


To reject: Play a falsifier (correct) 
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Using Pumping Lemma 


Play a falsifier: This is how this lemma is used. 

* To show that 0”1” is not regular 
* Anticipate any ny 21 
* Choose 0"1” € L such that 2n > ny 
« Anticipate any decomposition w = xyz: 

— Case 1:0¢ 0 1" (b> 1) 

= Case 2:0" 1¢ 14(e21) 

— Case 3: 0* (0'1°) 1¢(b+c2 1) 
* Choose i=2 
* 0071" ¢ L, 0114 ¢ L, and 04(0°1°)714 € L 
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// End 
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Pu m pi ng Le mM m a (for regular languages) 


To show that a language is nof regular 


For any infinite regular language L, 

there exists a positive integer n, such that 
for any w € L such that |w| = 1), 

there exists a decomposition w = xyz where 
|xy| < np and |y| = 1 such that 

for any i= 0, 

xyize L 
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w$w* is not regular 


To show that w$w* is not regular Play a falsifier 
Anticipate any ny = 1 

Choose w$w¥ € L such that |w$w4| > no 
Anticipate any decomposition w$w* = xyz where 
|xy| Sp and |y| 2 1: 

— Case 1 (y includes $): w, (w,$w,*) w,¥ [lw,| 20] 

— Case 2 (y does not include $): w, w, (w$w*) 
Choose i = 2 

w(wo$w*)w) € L, wyw22w3$wk € L 


CSC460 C5 


Unit C5: Overview 


¢ Distinguish CF and non-CF languages 
— Pumping Lemma (for CF languages) 

¢ Analyze the effects of combining 
subproblems (regular or CF) 
— Closure properties 

¢ Wrap up Module C 
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Using Pumping Lemma 


Play a falsifier: This is how this lemma is used. 
To show that 0"1” is not regular 
Anticipate any ny = 1 
Choose 0"1" € L such that 2n > ny 
Anticipate any decomposition w = xyz: 
— Case 1:0 0 1"(b21) 
— Case 2:0" 1° 14(c21) 
— Case 3:0 (01°) 1¢(b+c 21) 
Choose i = 2 
020251" ¢ L, 0"12°14 € L, and 09(0°1°)714 ¢ L 
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ww is not regular 


To show that ww¥ is not regular Play a falsifier 
Anticipate any ny = 1 
Choose ww® € L such that ww’ = 0*1uu®10* where 
k> hy 
Anticipate any decomposition ww* = xyz where 
and |y| 21: 

— 0? 04 (O'luu® 10°) where p+ q+r=kandq21 

* Choose i =2 

© 07(07)?0"luuR10k ¢ L 
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Context-Free? [general] 


HTML/XML 

Processing recursive function calls 
Depth-first tree traverse 

Overhauling (repairing a physical device) 
Sample Problem #20 “respectively” 
Dishwashing 

Teaching a course 
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Context-Free? [formal] 


Or12 

Omir" 

{ww'w | w® is the reverse of we {0, 1}*} 
{ww|we {0, 1}*} 
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Main Property of CFLs 


Parenthesis matching 

Use of a single stack 
Arbitrary nesting is possible 
=O, le, Fi 
—0"2"1", 072"3"1" 

—OK2"00 17, OF2"3"00-—H 1" 
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0"1"2" is not CF 


To show that 0"1”2” is not CF Play a falsifier 
Anticipate any n, 2 1 

Choose 0"1"2" € L such that |0"1"2"| > Bip 
Anticipate any decomposition 0"1"2" = abcde 


where |béd| <7 and |ba| > 1: 

— Case 1: bcd spans part of 0"1" 

— Case 2: bed spans part of 12” 

Choose i =2 

At least one of 0 and 1 is pumped: i.e., 07+!172” 
or 0" 1712” or Ort! nt!2" 
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Analysis of CF-ness 


* To Show that L is CF 
— Proof by existence: Give a CFG or PDA 
* To Show that L is not CF 


— Need to prove that no CFG (or PDA) can 
generate the language 


— Demonstrate that some property of CFLs 
cannot hold 
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Pumping Lemma (or cris) 


To show that a language is not CF 
For any CFL L, 
there exists a positive integer n, such that 
for any we L such that |w| 2 1, 


there exists a decomposition w = abcde 
where |bcd| < ny and |bd| = 1 such that 


for any i= 0, 
* abicdie € L 
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Group Exercise 1 


Context-free? Justify. 
A. {O'2|i= jf +k 
B. {O'V2k |i < j <k} 
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Combining Subproblems Dealing with Subproblems 


* Compilers 
— Lexer: Regular 


Serial connection ~ Concatenation 
Parallel/alternative connection ~ Union 
— Parser: Context-free 


Repetition ~ Kleene closure 
— Semantic analysis: Decidable (ignore the : ; 7 : 
eeniesissensitve level) Simultaneous requirements ~ Intersection 


— Compiler as a whole: ? Negation ~ Complement 
* Bird flocking 

— Each bird (per frame): Regular 

— Entire simulation: ? 
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Closure Closure: Regular 


* Aclass C is Closed under binary operation Concatenation 
op. Union 
— The result of computing C, op C,, where C,, C, Kleene closure 
€ C, is still in C. C t 
: ; omplemen 
* Aclass C is Closed under unary operation P 
op. Intersection 


— The result of computing op C,, where C,; € C, 
is still in C. 


CSC460 C5 CSC460 C5 


Closure: Context-Free Group Exercise 2 


Concatenation 


CFLs closed under the following operations? 
Union 


A. Intersection 
Kleene closure B. Complement 


Intersection Een 


Complement 
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Intersection of CFLs Unit Summary 


‘Lys 0712! e CFL * CF vs. non-CF 

4 — To show CF: Give a CFG or PDA 
: L, = 01172" € CFL — To show not CF: Use the Pumping Lemma 
‘LAL 012" ¢ CFL * Closure: Regular languages 


— Closed under concatenation, union, Kleene closure, 
complement, and intersection 


* Closure: CFLs 
— Closed under concatenation, union, and Kleene 


— Not closed under complement or intersection 
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Extended Chomsky Hierarchy 


Examples 


« Pumping 

— Gas (automobile) 

— Air (human lung) 
* Computer environment 

— Occurrence of arbitrary events 

~~ CFG/PDA — Modularity, recursion, interrupt, etc. 

Ce gene (DPDA * Baby (newborn) 
: ee TERIOR — Limb movement 


— Remember things after sleep? 
CSC460 C5 CSC460 C5 


TM-recognizable (RE) *-- Grammar/TM 


Decidable (Recursive)*<----\-----_ /Decider 


Context-Sensitive CSG/LBA 


Module C Summary 


* Chomsky hierarchy and properties 


— Regular languages: mostly closed and 
extremely fast to process 


— CFLs: partially closed and potential 
processing drawbacks 


* E.g., compilers only deal with DCFLs 
— Beyond: more closed but easily undecidable 
* Distinguishing classes 
— Constructive vs. Pumping Lemma 
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Unit C5 Supplement, 3/29/05 


Slide 4: ww* is not regular 

After looking at the slide again, I realized that the slide is correct. The key is to choose ww* = 
O‘1uu* 10 where k > no. This way, xy falls on the first 0“. By pumping y (e.g., y*), we introduce 
additional 0’s, which will conflict with the 1 preceding the last of. 


Slide 12: Group Exercise 1 Problem A. {0'1/2* |i= j +k} 


The question was as follows. By choosing w in a way bcd <i, the falsifier could show that any 
decomposition of w would destroy the context-freeness. Wouldn’t it suggest that the language is 
not context-free? 


The Pumping Lemma for CFLs is slightly different from the version for regular languages in that 
the span of bcd can move around more flexibly due to the lack of constraints on a. So, for this 


case, the verifier will be able to slide bcd so that b and d would fall on parts of 0 and 1, 
respectively, to allow arbitrary pumping. 


// End 
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Name: 


Exercise C5 (Module C Comprehensive Exercise), 3/29/05 


Note: This exercise is announced in advance so that we can think about it through this module. 
The due date of this exercise is the date of Module C Evaluation Workshop. 


Part 1: Identifying the Simplest Mechanism Along the Extended 
Chomsky Hierarchy 


One of the main goals of Modules C is to be able to identify the minimal mechanisms for real- 
world problems (so that we won’t use insufficient or overly expensive models). In this module, 
we analyze several representative problems, e.g., lexer, parser, and variable co-reference in a 
compiler. The real test for us is whether we can develop and describe our thought process 
involved in such activities and apply it to a new problem. While this is another major problem in 
the Theory of Computation, it is unlikely that you will easily find an answer in the literature. By 
all means, it is a challenge. So, you are not expected to come up with a rock solid answer, not to 
mention a correct answer. Do your best. As usual, you are encouraged to discuss with other 
students and the instructor (but your writing must be your own, reflecting your own thought, 
which must be unique to yourself). If we as a class come up with useful analytical tools, it 
would be great. 


Note: Since Tasks 1 & 2 are closely related, you should tackle them in parallel, or in a circular 
manner (back and forth). 


Task 1: Develop your own “heuristics” to analyze a given problem with respect to the minimal 
mechanism, 1.e., decidable, context-free, deterministic context-free, regular 
grammar/automaton/language. By “heuristics,” we mean a speculative, but reasonable method 
of analyzing any given problem. 


Task 2: This task is an application of the previous one. 


(A) First, identify a new, unique practical problem (in or outside computer science) which would 
be a good example of demonstrating your heuristics in Task 1. Then, identify multiple 
interactive subproblems/modules of your problem. If the problem is complex, you can focus on 
the areas which you are most interested in. But still try to include modules of different 
complexity. Also, try to represent the system schematically. 


(B) Next, for each subproblem/module, 

1. Analyze its task and clearly present it as a problem (if possible, as a computational problem 
in the set notation), 

2. Applying your heuristics developed in Task 1, identify the class of the minimal grammar or 
automaton that would correspond to the problem (mainly consider the extended Chomsky 
hierarchy: 1.e., decidable, context-free, deterministic context-free, or regular), and justify 
your choice, and 

3. Describe the component’s specification or operation at a reasonable level. 


1/2 


CSC460 


Part 2: Evaluation Form and Supporting Notes 


Review the evaluation procedure. Then, complete your evaluation form and supporting notes. 
Bring them to the evaluation workshop (hard copy). Print them well in advance so that you can 
avoid potential problems, e.g., not being able to print just before the evaluation. 


Survey: Time spent between after Unit C5 before the evaluation workshop: 


// End 
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Module C Review Exercise, 4/1/05 


This exercise is to be done in groups of two students (unless otherwise directed) assigned at this 
time. You need to complete the exercise within 30 minutes, but at the same time, you need to 
use up the time so that you can come up with a response as informative and complete as possible. 
While both of you in a group must agree on the general ideas, you must write your own version 
of the response. That is, your writing must be in your own words and not a word-by-word copy 
of your partner. Now, choose one of the options listed below. If your example problem in the 
comprehensive exercise and/or mini research problem(s) is/are similar to one of these, avoid 
discussing the same subject. For example, if you analyzed vision or mind as your mini research 
problem, you can still choose Option 1, but focus on different aspects of the system. 


Option 1: Human Being/Cat/Robot 


First, analyze the functionality of a human being, cat, or robot (e.g., sensation/perception, 
cognition, emotion, behavior, physiology, kinesiology, etc.). 


Option 2: Operating System 
First, analyze the functionality of a hypothetical operating system (not including applications 
such as compilers, which have been discussed frequently). 


Common procedure 


Then, construct a modular system consisting of multiple interactive components. You can focus 
on the areas which you are most interested in; there will be no time to cover the entire system. 
But still try to include modules of different complexity. Also, try to represent the system 
schematically (i.e., with a diagram). 


Next, for each component, 

1. Analyze its task and clearly describe it as a computational problem, i.e., a language (if 
possible, in the set notation), 

2. Using your heuristics, identify the class of minimal grammars or automata that would 
correspond to the problem (mainly consider the extended Chomsky hierarchy: non-TM- 
recognizable, TM-recognizable, decidable, context-free, deterministic context-free, regular; 
exclude context-sensitive), and justify your choice, 

3. Describe the component’s specification/operation at a reasonable level, and 

4. [time permitting] Analyze the interface with other interacting modules, e.g., by comparing 
the input and the output of the interacting components. 


Be prepared to informally present your response to the class [approx. 5 minutes per 
group]. 


// End 
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Exercise C6/D0, 4/1/05 


Complexity (Data-Size Scalability) 


During the first half of Module D, we will be discussing the third subarea of the traditional 
Theory of Computation, the efficiency of an algorithm with respect to the input size (data-size 
scalability). For example, you must remember that while a naive sorting algorithm would run in 
O(n’), “quick sort” runs in O(n log n). You must be familiar with this subject to some extent 
from other courses, most notably, Advanced Algorithms. So, try to bring in whatever you know 
and expand on it. As in Module C, I often notice that students’ understanding of this area varies 
greatly. As we proceed, try to exchange what you know with other students and the instructor so 
that we all learn through the process. 


Task: Write a concise essay about algorithm efficiency with respect to the input size (of course, 
for a decidable problem/subproblem), referring to (i) your problem from Exercise 00 and (11) 
your mini research question(s) from earlier exercises or different one(s) from the list (below). In 
addition to discussing these problems in their entirety, you must also consider possible 
subproblems. 


Note: If possible, it would be good to provide a precise asymptotic analysis, e.g., the O notation 
(or its variants). However, if you are not comfortable doing so, try your best to describe (1) how 
the algorithm performance depends on the input data size and (2) the difficulty you are having 
with asymptotic analysis. In any case, you may want to review the materials in Advanced 
Algorithms. 


List of sample research questions from Exercise A6/B0 

Can organizational dynamics be modeled as an algorithm? 

Can evolution be modeled as an algorithm? 

Can ecology be modeled as an algorithm? 

Can human development be modeled as a computer? 

Can our minds be modeled as a computer? 

Can vision be modeled as an algorithm? 

Can learning be modeled as an algorithm? 

What would be the minimal mechanism to process human language? 
Can the entire situation of an arbitrary game be modeled as an algorithm? 
10. Would “perfect” user modeling, e.g., for web search, be possible? 

11. Would “perfect” computer security be possible? 

12. Can the entire process of software engineering be modeled computationally? 
13. Can computer networks be modeled as a single computer? 

14. Could biology be reduced to physics? 

15. Could some computer generate real (not pseudo) random numbers? 

16. Would it be possible to decide whether the given numbers are random? 
17. Would randomization affect computability and/or complexity? 

18. Would parallelism affect computability and/or complexity? 

19. Would artificial neural network be more powerful than TMs? 

20. Would cellular automata be more powerful than TMs? 

21. Would the use of analog (or fuzzy) values affect computability? 

22. Would relativistic, quantum, or some other modern-physics-based computation surpass TMs? 


SONORAN 
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23. 


Can all the cases of on-line algorithms be simulated by off-line computation? [On-line algorithms would 
obtain inputs as the time progress. Off-line computation would provide all the possibilities as input at once. Cf. 
function-to-relation conversion used to fold the output within the input] 


. Would oracle computing affect computability? [Oracle computing: A TM with the capability to ask questions 


to another mechanism] 


. Would persistent TM be able to compute more than the standard TM? [Persistent TM: Multiple sessions of 


TM operation with some memory between them] 


. Would accelerating a TM give more power? 

. Would slight error tolerance affect any aspect of the Theory of Computation? 

. What would be the ability of a finite automaton with a queue? 

. What would be the effect of “constant” (as in complexity analysis) in practice? 
. Can any mathematical function be represented computationally? 

. What exactly are power sets doing to the Theory of Computation? 

. Is what you can do in logic the same as what you can do with computation? 

. If you have a research question of your own, please consult the instructor first. 


Survey: Time spent between classes: 


// End 
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Unit C6 Supplement, 4/4/05 


Language and the Set Representation of a Problem 

After reading some (but not all) of your Module C Comprehensive Exercises, I noticed that some 
of you confused the notion of “language” and “problem.” Probably, I did not emphasize this 
point sufficiently. So, here is another supplement to clarify the connection. A language (set of 
strings) can always be seen as a problem (naturally, a set representation). For example, 0"1" = 
{0"1" | is a natural number} is a language and a problem at the same time. However, not all 
problems in set notation can be a language. For example, a problem ACCEPT; = {(M, w) | TM 
M accepts string w} is not a language because a member of this problem is a “pair,” not a string 
(even though the pair contains a string). Some problems would be represented as a set of more 
complex mathematical structures (“tuples” of various components). To analyze problems with 
respect to our extended Chomsky hierarchy, we must represent problems not just as a set, but as 
a language, because that is what grammars and automata are supposed to specify/process. 


In some cases, it would be more natural to represent a problem using a mathematical structure as 
in ACCEPT; y. In many cases, we can still modify the problem as a language. For example, by 
considering the string representation of the formal definition of a TM, we might represent 
ACCEPT; as {M#w | TM corresponding to the formal definition M accepts string w}, which 
can be processed by a TM. Another example would be binary addition. We can represent it as a 
problem (but not language), {(x, y, z) |x + y =z, where x, y, z are non-negative binary numbers 
(values) }, or a language/problem, {x#y#z | val(x)+ val(y)= val(z), where x, y, z are non-negative 
binary numbers (strings) whose values are evaluated with the function val}, which can be 
processed by a TM (or possibly some other automata). 


// End 
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Module C Evaluation 


Review your portfolio 

« My comments are sporadic and scattered 
on Review Ex, Comprehensive Ex, Supp. 
Notes, and Reflective Essay. 

« You are encouraged to clarify and discuss 
my comments. 

* You can keep the folder till the next class; 
then, return it to me. 
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Module D Plan 


Polynomial vs. Exponential 
— Intractability 
— Nondeterministic Polynomial-time (NP) 


D2 NP-Completeness (NPC) Mini research. 


D3 Space complexity 
D4 Parallel computation 
Super-Turing computation Reading 
D7 Evaluation workshop 
YY Practicum evaluation (Wed., Apr. 27) 
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Content Goal 3: Interactivity 


. Understood the effects/limitations of parallel 
computation with respect to the three subareas of the 
traditional Theory of Computation. 

. Understood what kind of problems cannot be 
adequately represented by TMs. 

. Understood the basics of super-Turing computation 
(more “powerful” than TMs) including its significance. 

. Was able to speculate where the theoretical 
underpinning of computer science should be heading, in 
order to offer robust analyses of a variety of 
computational problems. 
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Overview: Theory of Computation 


impossibility |Computability 


Uninteractable? Focus in Module D 


Interactable? 
EERE 
Distributivity? | Undecidable 
= 

DD 


cidable 


real practicality 


z SS 
Intractable Context-free 
r _ me 
Complexity] Tractable Regular 


- /~ 
practicality | Languages’ simplicity 
Automata 3 
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Content Goal 6: Complexity 


. Reviewed how to interpret the big O notation [game-theoretically, using the 
on-line graphing tool]. 

. Understood that exponential growth with respect to the input data size (time 
complexity) is considered impractical (“intractable”), through examples. 

. Understood the class of “nondeterministic polynomial” (NP) problems, 
through examples. Also understood (i) why there are so many NP problems 
and (ii) why NP problems are essential for computer security. 

. Understood the notion of “polynomial time reducibility” and its impact on 
relating problems. 

. Understood the class of “nondeterministic polynomial complete” (NPC) 
problems, through examples. Also understood the basics of showing that a 
problem is in NPC. 

. Understood the essential difference between time and space complexity, as 
well as the hierarchy of time/space complexity. 

. Could explain (i.e., teach) this goal to CS students outside this class. 
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Unit D1: Overview 


Review basic ideas about algorithm 
analysis 

Identify the gap between practical and 
impractical algorithms 

Explore some questionable cases with 
respect to practicality 


Preview Exercise D1 “Complexity 
Analysis” 
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How to Choose Particles in Japanese? 
(in English-Japanese Translation) 


Title: Osteoporosis in Active Women 


(i) English: | Osteoporosis is a disease of bone. 


Japanese’: Osteoporosisiwa| bone-of disease-is. 


(ii) English: | Young females are most affected. 


Japanese’: Young femaletga] most are.affected. 


sateen tcc 9S BERR ep aan tea ace 
i) ° 


| 


Without spurious ambiguity elimination 


With spurious 
Peal teh ee 
ambiguity elimination 


380g 
rin Geedbo” p28" °° 


= ne 


es 
10 20 30 40 50 [words] 60 


Figure 2: English Parser with and without Spurious Ambiguity Elimination 


Digital Circuits (Sample Problem #13) 


* To design digital logic circuits, we need to 
analyze the relation between inputs and the 
output. For example, the following formula 
represents a circuit with four inputs and one 
output, each of which could take either 0 or 
1 (‘+ for OR, ‘’ for AND, and ‘” for NOT). 
output = (7, +i, +i, +i,)-@'+it+i,4+i,)-@’ 
+h FL +1,) 

¢ Brute-force algorithm? 
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Parsing English Texts 


* Treating Hypertension in Active Patients: Which Agents 
Work Best With Exercise? 


Hypertension is one of the leading risk factors for 
cardiovascular morbidity and mortality. Unfortunately, the 
percentage of those who have hypertension may be on 
the rise as more and more Americans remain sedentary. 
Physicians, therefore, need to promote exercise 
programs (see “Exercisers: A Healthy Minority,” below)— 
but without unwittingly sabotaging exercise capacity with 
antihypertensive medications. So in addition to knowing 
antihypertensive agents’ impact on exercise, clinicians 
need to target therapy for varied hypertensive patients 
who participate in different sports and types of exercise. 
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Practicality 


« Where can we draw a line between 
practical and impractical algorithms, esp. 
facing realistically large data? 


* Focus on the effect of input data size 
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Cross-Country Interview (Sample Problem #16) 


* Suppose that you are invited for interviews 
(graduate schools or industry jobs) ina 
number of cities in the US. Unfortunately, 
they do not pay for your travel. So, you 
will need to minimize the expense. 


¢ Simple-minded algorithm? 
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Faster Algorithms 


* Count from 0 to the user-specified n 
n <- input 
for (int i = 0; i <n; ++i) { 
output <- i; 


} 
¢ Display elements of a n x m matrix 


n <- input; m <- input; // set elements in matrix 
for (int i = 0; i <n; ++i) { 
for (int j = 0; 3 < m; ++3) { 
output <- matrix[i, j]; 
} 
} 
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Big-O Notation 


* Informal idea: Behaves roughly like such 


an such function popular use of = instead of ‘=* 
* Definition #1: f(x) € O(g(n)) if there are 
constants c > 0 and ny, 2 1 such that 
fin) $c g(n) for every integer n = ny. 
* Definition #2: 
f(n)e O(g(n)) if tim) =c forsomec =0 


g(n) 
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Common Time Complexity Classes 


* O(log n) = Logarithmic 

* O(n) = Linear 

* O(n log n) 

* O(n?) = Quadratic 

* O(n?) = Cubic 

* Note: O(n') Cc O(n’) where i <j 
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Time Complexity 


(asymptotic analysis) 

* The speed/time performance of an 
algorithm as a function of the input data 
size, ignoring the constant factor 

* Can be analyzed with the big-O notation 
(and its variants) 
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Group Exercise 1 


A. Explain why n? € O(2”) 
B. Explain why 2” ¢ O(n) 


* Definition #1: fin) € O(g(n)) if there are 
constants c > 0 and ny 2 1 such that fin) <c 
g(n) for every integer n = no. 

* Definition #2: 

f(nje O(g(n)) if lim Lu =c forsomec 20 
noo & n 
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Polynomial vs. Exponential 


* Polynomial-Time (Bound) Problems 


P = O(n) for any constant k > 1 
* Includes faster algorithms as well: O(log n) c 
O(n logn) cP 


* Exponential-Time (Bound) Problems 
Exp = O(c") for any constant c > 1 
+ Includes faster algorithms as well: P c Exp 
+ (strictly) Exponential problems: Exp — P 
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Intractability 


¢ Tractable problems = P 
* Intractable problems = Complement of P 


— Including exponential problems (brute-force 
algorithms are not acceptable except for toy 


examples) 


csc46o 01 Is the gap clear cut? 


SAT & TSP: Properties 


* Search space: Exponential 
« Exhaustive search = Exponential 
* Checking an answer => Linear 


* No polynomial algorithm has been found. 
=> Most likely intractable 


— But nobody has proved the nonexistence. > 
Could be tractable 


Practical approach => Approximate 
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Nondeterminism 


* Possibility of exponential branching 
* Does not change the power of TMs 


* Could change the time complexity of 
processing certain NP problems 


CSC460 D1 


19 


Sample Problems, Revisited 


* Digital Circuits = SATISFIABILITY (SAT) 
— Example: @ or q) and (p or (not g)) and ((not p) or r) 
— Search space: Exponential 
— Practical algorithm? 


* Cross-country interviews = TRAVELING 
SALESPERSON (TSP) 


— Search space: Exponential 
— Practical algorithm? 
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Approximate Solutions 


Classical local search (SAT, TSP) 
Greedy algorithm (SAT, TSP) 
Dynamic programming (TSP) 
Simulated annealing (SAT) 
Genetic algorithm (SAT, TSP) 
Neural network (TSP) 


How to Solve It: Modern Heuristics by Michalewicz and Fogel 
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Class NP 


¢ Class of Nondeterministic polynomial-time 
problems 


—E.g., SAT, TSP, and many others 


¢ Definition #1: A solution can be verified in 
polynomial time. 


¢ Definition #2: A nondeterministic TM can 


decide in polynomial time. 
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Group Exercise 2 


A. Show that a scheduling problem (find a 
combination that satisfy some constraint) 
is in NP, using the both definitions. 


B. Isa NP problem tractable? Explain? 
*Definition #1: A solution can be verified in polynomial time. 


*Definition #2: A nondeterministic TM can decide in 
polynomial time. 
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Unit Summary 


* Use of the big-O notation for analyzing 
time complexity 

¢ Hierarchy of time complexity classes: P 
(tractable), NP (open), Exp (intractable) 


— No polynomial algorithms have been found for 
any of the NP problems. 


-P=NP? 
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Sample NP Problems 


Cryptography (Sample Problem #11) 
Monkey Puzzle (#12) 


Professor Assignment (#14) ~ scheduling 
problem 


Knapsack Problem (#15) 
CPU Register Allocation (#17) 
Map Coloring (#18) 
Why so many? 
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Summary Question 


¢ Do you understand the definition of 
NP? 


—Don’t leave until your answer becomes 
yes. 


* Questions/Comments/Suggestions 
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Unit D1 Supplement, 4/8/05 


Sample Response to Exercise D1 Part 1: Big-O Notation 

This exercise is intended to serve not only as a review of algorithm analysis but also as a review 
of logic. The use of game-theoretic interpretation often reveals whether one understands the 
semantic aspect of logic (which is more important than the syntactic aspect, in my opinion). If 
one can associate quantifiers with the operations of the graphing tool, s/he clearly demonstrates 
how to deal with quantifiers. I realized that most of you need to see how to do it. So, here is 
how I would do. 


Definition: ‘O’ (asymptotic upper bound) is defined as follows: f(n) € O(g(n)) if there are 
constants c > 0 and no 2 1 such that f(1) < c g(n) for every integer n = no. 


The definition can be translated into the following first-order logic (FOL) statement (use of ‘A’ 
for ‘and’): 


Ac Ano Vn [ (c > 0) A (19 2 1) A (n= N90) A (f(1) ¥ € g(n)) J 


which can also be written as: dc > 0 dnp = 1 Vn = no [ f(r) < c g(n) J. As for the ordering of the 
quantifier, Ino and Vn must be placed in that order because the choice of n depends on that of no. 
The placement of Hc is flexible because only the main relation, f(7) < c g(n), depends on it. 


Advanced notes (can be skipped): Although it would also be possible to quantify the functions as follows: 
Vi Vg de Ang Vn [(c > ODA (19 2 DA (M2) A (F(T) Sc gn) J 


such a statement is beyond the ability of FOL because functions are “higher-order” and not individuals. 


The general procedure would be as follows. First, assume that the two functions are given. 
Then, define a strategy for each quantifier. 


1. Verifier: Choose c > 0 by adjusting the slide bar so that the statement would hold. 

2. Verifier: Choose no 2 | by finding the value by adjusting the scale(s) so that the statement 
would hold. 

3. Falsifier: Choose n = no by finding the value by adjusting the scale(s) so that the statement 
would not hold. 


If the falsifier can find such a value, the falsifier wins, i.e., the statement does not hold. 
Otherwise, the verifier wins, i.e., the statement holds. 


Example 1: n* € O(2") 


e Set f(n) = n'\, g(n) = 2" Draw the graph with x-axis: 101, y-axis: 103 (default); f(n) = n’ 
looks like growing faster. 
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The verifier keeps the default c = 1, but searches for a no by adjusting x-axis to 10%2. Since 
the values go beyond the current range, s/he increases the y-axis to 10“6. Since g(n) = 2” 
appears to exceed f(1) = n’ after n= 16 (or so), the verifier sets the no to 20 (to be safe). 
The falsifier tries to find an n 2 no at which the situation reverses by extending the x-axis 
(and y-axis to increase the range). However, s/he fails to find such an n. The falsifier lost. 
Thus, n* € O(2"). 


Example 1:2” ¢ O(n’) 


Set fin) = 2”, g(n) =n’ Draw the graph with x-axis: 101, y-axis: 1043 (default); f(n) = 2” 
looks like growing faster. 

The verifier tries to raise c = 10. Then, g(n) =n” appears be no less than f(n) = 2”, within the 
window. Just to be make c g(n) larger, s/he raises c to 10,000. S/he stays with no = 1. 

But then, the falsifier can adjust the scale to observe that f(n) = 2” exceed g(n) = nat some 
point. Although this step depends on the verifier’s choice of c and no, the falsifier seems to 
be able to find a point where f(n) = 2” exceed g(n) = n’, at least within the operations of the 
graphing tools. The falsifier won. Thus, 2” ¢ O(n’). 


// End 
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Name: 


Exercise D1, 4/5/05 
Part 1: Big-O Notation 


The big-O notation is useful for analyzing the effect of the input data size on the performance of 
an algorithm (not including the constant factor corresponding to, e.g., fixed amount of speed up). 
Although this concept must have been discussed in other courses, e.g., Advanced Algorithms, it 
would still be good to review the idea. To do so, we will re-use some of the tools introduced 
earlier in this course, 1.e., game-theoretic interpretation of FOL and the on-line graphing tool. 
First, here is a definition of the big-O notation. 


Definition: ‘O’ (asymptotic upper bound) is defined as follows: f(n) € O(g(n)) if there are 
constants c > 0 and no 2 1 such that f(n) < c g(n) for every integer n = no. 


Task: Give a game-theoretic interpretation of the above definition referring to the operation of 
the on-line graphing tool (http://www.tcnj.edu/~komagata/Graphing). Note that you can adjust 
the constant c using a slide bar (and a text field) and designate some no by simply identifying a 
specific value on the x axis (note that you can change the graph scale). 


Hint: If you are not sure how to use the tool, read the tutorial part of the documentation linked 
from the graphing tool page. 


Part 2: Sample Problems 


As discussed in class, there are a large number of NP problems. NP problems are important 
because many practical combinatory problems exhibit the properties of this class. 


Task: Choose at least two problems from the Sample Problems 


(http://www.tcnj.edu/~komagata/csc460/05s/SampleProblems.pdf) listed below, and informally 
explain why the problems are in the class NP. 


#11 Cryptography 

#12 Monkey Puzzle 

#15 Knapsack Problem 

#17 CPU Register Allocation 
#18 Map Coloring 


Survey: Time spent between classes: 


// End 
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Questions 


¢ How good is computer security? 


¢ What would computer security 
professionals and hackers pursue? 
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Review 


NP Problems 


* Class of Nondeterministic Polynomial-time 
problems 
—E.g., SAT, TSP, and many others 


* Definition #1: A solution can be verified in 
polynomial time. 


* Definition #2: A nondeterministic TM can 


decide in polynomial time. (ee 
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Standard NP Problems 


* SATISFIABILITY (SAT) 
» 3SAT 

» TRAVELING SALEPERSON (TSP) 
* HAMILTONIAN CIRCUIT (HC) 

» VERTEX COVER (VC) 

* CLIQUE 

* 3D MATCHING (3DM) 

* PARTITION 
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Unit D2: Overview 


* Notice the connection among NP 
problems 

¢ Understand polynomial time reducibility 

* Identify and analyze a special class of NP 
problems 

* Preview Exercise D2 “NP/NPC Practice; 
Mini Research Ideas” 
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Review 


Sample NP Problems 


Digital Circuits (#13) = SATISFIABILITY (SAT) 


Cross-country interviews (#16) = TRAVELING 
SALESPERSON (TSP) 


Cryptography (#11) 

Monkey Puzzle (#12) 
Professor Assignment (#14) 
Knapsack Problem (#15) 
CPU Register Allocation (#17) 
Map Coloring (#18) 
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SATISFIABILITY (SAT) 


* Informally, finding a satisfying truth-value 
assignment for a conjunction of 
(disjunctive) clauses, e.g., “lause 
(p or q) and (p or (not q)) and ((not p) or r or s). 

— Note: A clause is a disjunction of propositional 
(true/false) variables or its negation. 

* Formally, {(C, V, a) | set of (disjunctive) 
clauses C with a finite set of variables V 
such that assignment a: V > {T, F} 
satisfies all the clauses in C} 


3SAT 


* Aspecial case of SAT with the pattern 
(_ or _ or _) and (_ or _ or _) and... and (_ or _ or _) 
where each ‘_’ holds a variable (e.g., p) or 
the negation of a variable (e.g., not q) 
—l.e., each (...) is limited to 3 things 

* Connection to SAT? 
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HAMILTONIAN CIRCUIT (HC) 


* Informally, finding a closed circuit to travel all the 
points without repeating. 

* Formally, {G| graph G =(V, £) admits a 
hamiltonian circuit} 

* hamiltonian circuit: (v,, .... y.=)y) such that 
v,€ Vfor every 1 <i<k, 
v, # v; for every pair i #j, 
(vj; Via 1) € E for every 1 <i<k, 


pV) € E Hamiltonian Path ~ between two points 


* Connection to TSP? Euler circuit ~ traversing all the edges 
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CLIQUE 


¢ Informally, finding a subgraph of size k or 
less such that every two nodes are 
connected by an edge in the graph. 

¢ Formally, {(G , k) | graph G =(V, E£) 
contains a k-clique where 0< k<|V]} 

¢ k-clique: Wc V such that 
|WI=k, 
For every u,ve W,(u,v)e E 

* Connection to VC? 
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TRAVELING SALESPERSON (TSP) 


* Informally, finding a tour of all the cities (without 
repeating) within the set budget. 

+ Formally, {(G, c, b) | graph G = (V, E) with the 
cost function c such that a tour is possible within 
the total cost of b} 

© four: (Vy, 5 Meyyp such that 
v,€ Vfor every 1 <i<k, 
v, # v; for every pair ij, 
(v;, V;41) € E for every 1 <i<k, 


Owe k “completeness of 6? 
c: Vx V— Z* (positive integer) 
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VERTEX COVER (VC) 


* Informally, finding a graph segment of size 
k or less that contain at least one point of 
every edge. 

* Formally, {(G,, k)| graph G =(V, E) admits 
a vertex cover where 0< k<|V]} 

* vertex cover: Wc V such that 

|W|<k, 
For every (u, v)<¢ E,ue Worve W 
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Marriage Problem 


* Informally, finding a complete list of male- 
female pairs (without polygamy) 

* Formally, {((R,k)|R CM~xF contains a 
matching where M, F are disjoint and |M| = 
[Fl =k} 

« matching: P c R such that 

|P| =k, 
Vix, y), uve P (x #u)AQyF¥v)] 
(No two elements of P agree in any coordinate) 


eee be NP? 2 


3D MATCHING (38DM) PARTITION 


+ Informally, a variant of the marriage + Informally, finding a subset that divides a 
problem involving 3 different sexes. set evenly with respect to the members’ 
* Formally, ((R,k)|R CWxXx Ycontains a “prices.” 
matching where W, x, Y are disjoint and |W| Formally, {(A, s) | finite set A with a 
= |x] =|¥] =k} balanced partition} 
* matching: M c R such that balanced partition: B < A such that 
|M| =k, p:A->Z*  (“price” function) 
V(x, y, Z), (u,v, w) Ee M [x4#u)A(VF#v)A(z¥W)] ¥ pla)= ¥ rl ) 


(No two elements of M agree in any coordinate) = actaes) 
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Connection among Problems Polynomial Time Reducibility 


* Special cases ¢ Ais polynomial time reducible 
— 3SAT (clauses of size 3) is a special case of (P-reducible) to B. 


SAT — Ais reducible to B. 
— HC is a special case of TSP (with weight) — This can be done in polynomial time. 
¢ Similarity * Transfer of properties 
— VC and CLIQUE — Bhas a positive property = A has it too. 
* Can some problems be reduced to others? + Example: If Be P, Ac P. 
— A has a negative property = B has it too. 
+ Example: If A ¢ P, Bé P. 
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NP-Complete (NPC) Problems Cook-Levin Theorem 


* Definition * SAT is NP-complete. 
1. The problem is in NP. 1. SAT is in NP. 


2. Any NP problem can be P-reduced to that 2. Any NP problem can be P-reduced to SAT. 
problem. [i.e., can solve any NP problem] * Challenge 


* Generic NP problem representation 
@. NPC Most difficult NP problems e Terminology 


(p or q) and (p or (not q)) and ((not p) or r) 
Generalization of X-complete y—~ 


* P-complete, Turing-complete, Al-complete clause clause clause 
CSC460 D2 
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Cook-Levin Theorem 
Proof Outline 


* Lemma: Any NP problem can be P- 
reduced to SAT. 


* Components 


« An arbitrary polynomial time Nondeterministic 
TM (NTM) program (or verification sequence 
of an answer) can be reduced to SAT. 

+ Acceptable strings > True SAT statements 
+ Acceptable strings < True SAT statements 
* The reduction is in P (polynomial reduction). 


* Complexity analysis 
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Cook-Levin Theorem 


Reduction Idea 


Introduce propositional variables as follows: 
¢ Simulating states 

— TM is in state q, at time i > Q;; ,, 
¢ Simulating head movements 

— Head is at tape position j at time i > H, 
¢ Simulating tape symbols 

— Tape symbol of position j is s, at time i 


ics Poor bouded? 


i,j) 
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+ States > 0.4, 
+ Head movements > Hj, 


3SAT Statements 


+ Attime 0, TM is in its initial configuration. = Qo) A Hoo, A Soosy ® 
Soap 0A Scont§,-» ® Sono) 0 + & S,o,potyzhound.0) 
+ Attime i, TM 


+ Is in exactly one state = at least two & at most one > (Qj) v ... 
V Qeipoty-bouna)) 0 (Qe A Quy) A + Hor all j #/’] ‘all clauses? 
+ Moves to a next configuration defined by 5. 


* Case 1 (atj at time i): 


Tape symbols => Si; 4) 
‘x’ =‘and’, ‘V' =‘or', = ‘not’ 


(Hay A Qty A Sein) > Aeir1j+-movemeny) © ¥ Dirty) A OV 
Scary 
Note (Hg. A Qin) A Si)  Aiistarmovemeny) (TA ¥ Qin V 
Si.) V Ast jasome value)) 
+ Case 2 (not at j at time i): i.e., no change 
(COAG A Stijn) > Sista) 


Note ((S(.:4 8G.) > Sst jay) (Any ¥ WSeijay Y¥ Stent iy) 
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Cook-Levin Theorem 
Polynomial Time Conditions 


* Cannot spend exponential amount of time 
(cf. the input) 


* Cannot use an exponential amount of tape 
space 


* The limit to polynomial tape space does not 
guarantee P. But exponential space cannot 
be handled in P. 
* Cannot use an exponential number of 
symbols 
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Cook-Levin Theorem 
Imposing Logical Conditions 


* At time 0, TM is in its initial configuration. 
¢ At time i, TM 
— Is in exactly one state, ieee 
— Is at exactly one tape position, and 
— Reads exactly one tape symbol, and 
— Moves to the next configuration defined by 5. 


¢ Within a polynomially-bounded time, TM enters 


the final configuration. + States > Op» 


+ Head movements > H,; ;) 
* Tape symbols > Sj, ; 4) 
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Cook-Levin Theorem Summary 


SAT is NP-complete. 
1. SATisin NP. < Truth value evaluation 
2. Any NP problem can be P-reduced to SAT. 


«  Anarbitrary NTM program in P is polynomial 
time reducible to SAT. 


* Simulate states, head movements, and 
tape symbols as a logical condition 

* Construct the statement in polynomial 
time 
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3SAT is NPC 


¢ 3SAT is in NP. 

* SAT is polynomially reducible to 3SAT. 
— If 3SAT can solve SAT, 3SAT is NPC. 

¢ Reduction idea 


— Transform each clause into an equivalent 
collection of clauses with 3 variables 

— Computability: Existence of an algorithm 

— Tractability: Polynomial bound 


phase ‘How about 2AT? 


25 


3SAT to VC 
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NPC Summary 


: 
E> ae SAT | Cook-Levin Theorem 
Coy ! 


3SAT VC 


ce 
He 


CLIQUE 


TSP 
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VC is NPC 


* 3SAT is polynomially reducible to VC. 


* Construct a graph with some & that would 
translate 3SAT to VC 


¢ Example 
(p or (not r) or (not s)) and (p or q or (not s)) 
(pv -=rv -s)A (VV qv 7s) 


* Polynomial reduction to CLIQUE possible 
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HC is NPC 


* VC is polynomially reducible to HC. 
* ... a bit complicated... 
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NP-Hard Problems 


* Definition 
— Any NP problem can be reduced to the 
problem. [at least as difficult as NPC] 


— Problem is not necessarily in NP. [OK to be in 
NP = NP-hard includes NPC] 


X-Hard for any class X 
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Unit Summary Summary Question 


* Awhole bunch of practical problems are in ¢ Do you understand the significance of 
NPC. NPC problems? Explain. 


* All NPC problems are equivalent with * Questions/Comments/Suggestions 
respect to their time complexity via 


polynomial reduction. 


* Apolynomial solution to any of these NPC 
problem would conclude P = NP. 
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Name: 


Exercise D2, 4/8/05 
Part 1: MAX-CLIQUE 


In the future, you will encounter new problems. In many cases, you will not know what the 
status of the problem is (unlike many school examples). So, it’s good to be prepared for such a 
situation. Here is a new problem that is related to CLIQUE. 


Let MAX-CLIQUE = {(G, &) | the largest clique of a graph G = (V, E) has k vertices}. Whether 
this problem is in NP is unknown. 


A. Explain what the underlined statement above mean. 
B. Speculate whether this problem is in NP. Try to justify. 


Part 2: 3COLOR 


Knowing that a problem is in NPC is a mixed message. However, with that information, we can 
be prepared to deal with the problem. Until there is a conclusion, we need to live with 
impractical exhaustive search or some practical approximate algorithms. 


Let us define a problem 3COLOR as {G| the nodes of a graph G = (V, E) can be colored with 
three colors such that no two nodes joined by an edge have the same color}. 


A. Show that 3COLOR is in NP. 

B. 3COLOR is actually in NPC. Explain how you would show that it’s in NPC. Note that you 
do not need to prove that 3COLOR is in NPC. However, you must identify all the necessary 
information that you would need to prove it. 


Part 3: Mini Research Ideas 


As the first step to tackle Module D Comprehensive Exercise, you will narrow down on the topic 
and specific research question. 


Task: Read the Module D Comprehensive Exercise carefully. Then, identify a mini research 
question which is related to the content of this course. If you feel that you have adequately 
solved a problem in earlier exercises, come up with a new problem to which you do not know the 
answer. 


Survey: Time spent between classes: 


// End 
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Code Talker & Chomsky 


« Who are “code talkers”? 
¢ What would Chomsky say about them? 
« Anything to do with time complexity? 
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Mini Research Ideas 
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Time vs. Space 


They are not so different [paraphrase of 
Einstein]. 

Time can be interpreted in another domain 
[applying Fourier transformation]. 

One cannot be certain about both position 
(space) aNd MOomMentuM (time dimension involved 
... velocity) [Heisenberg’s Uncertainty Principle]. 
Computers run faster with more memory 
[computer industry]. 
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Ex C2 


Part 1: MAX-CLIQUE 
* {(G,k)| the largest clique of G has k vertices} 
« Explain: “Unknown if it is in NP.” 
* Speculate 

Part 2: 3COLOR 


{G | the nodes of G = (V, E) can be distinctively 
colored with three colors} 


3COLOR € NP 
How to show 3COLOR e€ NPC? 
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Unit D3: Overview 


* Compare time and space complexity 


« Analyze the effect of nondeterminism with 
respect to space complexity 


¢ Preview Exercise D3 “Mini Research 
Paper Outline” 


— Different types of exercises from this time on 
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Time vs. Space Complexity 


* Time complexity 


— Maximum number of TM steps as a function 
of the input size (asymptotically) 


Space complexity 


— Maximum number of TM tape space as a 
function of the input size (asymptotically) 
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Space Complexity Hierarchy > 
Constant space 
— Any use? 
LogSPACE (also called L) — Usual use of a stack 

— Handling inputs ... isn’t it at least linear? * ALogSPACE algorithm possible? 
LinearSPACE 

— SAT (an exhaustive algorithm) 


Polynomial space (PSPACE) 
— Examples? 
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* Review: Time complexity? 
* Naive algorithm: LinSPACE 
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QBF Problem MAZE 


Quantified Boolean Formula * Finding a path between two points 
For all x, some y ((x or (not y)) and ((not x) or y)) — Also known as graph-reachability problem 
Vx dy (& v my) A (x v y)) ‘Satisfiable?- * Review: Time complexity? 
— Where each Variable is enier ne or false * Space complexity? 
— Cf. game-theoretic interpretation of FOL 
QBF Problem: { @ | @ is a true Quantified 
Boolean formula} 

* Review: Time complexity? 

* Space complexity? 
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Nondeterministic Space Complexity Hierarchy 


* Nondeterministic space complexity 


— Maximum number of NTM tape space as a 
function of the input size ae NPSPACE 


— Space measure for the successful run Y QBF 
| MAZE 


2 


¢ NLogSPACE (also called NL) PSPACE 


pe sve 
- NPSPACE Cas oe NLogSPACE 
LogSPACE 


— Example? 


Time 
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Today’s Questions 


1. Xvs. XSPACE 
E.g., P vs. PSPACE, NP vs. NPSPACE 
2. XSPACE vs. NXSPACE 


E.g., PSPACE vs. NPSPACE, LogSPACE 
vs. NLogSPACE 


3. Xvs. NLog(X)SPACE 
E.g., Exp vs. NPSPACE, P vs. NLogSPACE 
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Complexity Hierarchy 


Exp oo Crucial factor: Non-reusability of time 
(at least in a classic sense of time) 


joo NPSPACE 


eee NP tne 


pos PSPACE 
Pte 


LogSPACE 


Time Space 


v 
NLogSPACE 
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Effects of Nondeterminism 


Observation so far 
* L(TM) = L(NTM) 
* L(DFA) = L(NFA) 
+ L(DPDA) c L(PDA) |PDASaré nondeterministic! 
* PcNP 
* Probably P < NP (i.e., P # NP) 
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Group Exercise: Question 1 


X vs. XSPACE 
* E.g., P vs. PSPACE, NP vs. NPSPACE 
Which would be correct? Why? 

. Xc XSPACE 

. XC XSPACE 

. X= XSPACE 

. XD XSPACE 

. XD XSPACE 
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Group Exercise: Question 2 


XSPACE vs. NXSPACE 


* E.g., PSPACE vs. NPSPACE, LogSPACE 
vs. NLogSPACE 


Which would be correct? Why? 
A. XSPACE c NXSPACE 
B. XSPACE = NXSPACE 
C. XSPACE > NXSPACE 
D. None of the above 
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Group Exercise: Question 3 


X vs. NLog(X)SPACE 
* E.g., Exp vs. (N)PSPACE, P vs. 
NLogSPACE 
Which would be correct? Why? 
. XcNLog(X)SPACE 
. X=NLog(X)SPACE 
. X2>NLog(X)SPACE 
. None of the above 
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PSPACE-Complete 


* Definition: PSPACE-complete 
1. The problem is in PSPACE. 


2. Every PSPACE problem is polynomial time 
reducible to the problem. 


* Theorem: QBF is PSPACE-complete. 
* Proof idea 
1. Existence of a PSPACE algorithm 


2. Simulate polynomial-time TM (with a technique 
similar to the proof of Savitch’s Theorem) 
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Summary Question 


¢ What is your sense of time-space 
tradeoff, in the context of your choice 
(CS or else)? Discuss freely, 
referring to some example. 


* Questions/Comments/Suggestions 
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Unit Summary 


* Time and space complexities have some 
connection. 


¢ The effect of nondeterminism differ in each 
case. 


¢ Applications of complexity theory 


— NPC and beyond => Use of approximation 
algorithms 


— Effect of limited resources (of different types) 
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Unit D3 Supplement, 4/13/05 
QBF Problem 


The quantified variables in a QBF statement, 1.e., x, y, z, ..., can refer to any propositional 
variables. But since a propositional variable can take either true or false, we can analyze the 
truth-value assignment of a QBF statement with respect to possible values of true or false. Let us 
consider the example introduced in class: 


For all x, some y ((x or (not y)) and ((not x) or y)) 
[formally] Vx Sy (x v my) A (> Vv y)) 


To check Vx, we will need to consider both x = true and x = false. Let us first set x = true. Then, 
if we choose y = true, the entire statement can be made true. Next, let x = false. Then, we can 
choose y = false to make the entire statement true. Thus, this statement holds. The strategy of 
choosing values is analogous to the game-theoretic interpretation of FOL (although in FOL, we 
choose an individual or value in a more general sense, not true/false). 


When all the quantifiers are existential and the rest of the statement is organized as a conjunction 
of (disjunctive) clauses, the problem degenerates to SAT. So, if we can solve QBF, we can solve 
SAT. Thus, SAT reduces to QBF. 


QBF is NP-hard (i.e., it can solve any NP problem). SAT can be reduced to QBF. Since SAT 
is in NPC, QBF must at least as difficult as SAT. So, any problem that SAT can solve must be 
solved by QBF. Note that the (positive) property that SAT is in NP does not transfer to QBF. 


However, in the worst case, all the quantifiers can be universal. Then, we need to check all the 
combinations of truth values. So, even to verify an answer, it will take exponential time, as 
shown in an example with 3 variables below. 


QD 
S 
Ss 
io) 


CO] YI ON} (iy BIG] RO] ee 
A) |} ot) a] aya] 
td} a} 4a} ai) 4a] als 
Alay ayaa 


QBF is not in NP. Due to the quantifiers (the universal quantifier in particular), it is not 
possible to verify the truth value of a given statement in polynomial time. That is, even to verify 
a correct answer, we will still need to check all the combinations of truth-value assignment. 
Thus, it is worse than NP. 
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QBF is in PSPACE. Given any QBF statement, we can analyze its truth value in a way 
analogous to the analysis of the example statement above. Suppose that there are three 
universally quantified variables, x;, x2, and x3. First, set x; = true in the working tape position 1. 
Then, set x2 = true in the working tape position 2, and so on, as shown schematically below. 


X1 X2 X3 


If this assignment result in false, the entire statement cannot be true (because all the 
combinations must result in true). Using a recursion, we can cover all the combinations. So, we 
need O(n) working tape space, which is Polynomial. 


Time-Space Tradeoff 

One natural idea about time-space tradeoff may be that with more space, one can reduce the 
time. Scott (in his summary question) represented this (very roughly) as: Cost = Time x Space. 
Consider the following example. You open a computer repair shop in your own dorm room. If 
ten customers bring in ten computers, your work efficiency may well suffer from the lack of 
space. On the other hand, if you had a typical classroom, you could do various things much 
more efficiently. The time-space tradeoff for computers is often discussed along this line. 


Departing from this particular task, let us compare your dorm room and a typical classroom with 
respect to a broader range of tasks. If you need to vacuum, you will need more time for the 
classroom. If you need to keep the space warm, it will cost more to warm up the entire 
classroom. The discussion of the relation P C PSPACE appears to be more related to this view. 
That is, with more space, you could potentially do more things, which would potentially take 
more time. 


// End 
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Name: 


Exercise D3, 4/12/05 


Mini Research Outline 


Starting from the ideas you brought in Exercise D2 Part 3, you will develop the backbone of your 
mini research paper. The most important part at this stage is to identify your research question. 
Then, analyze its cost/significance and speculate how you would respond to the question. At that 
point, you may well revisit and revise your research question. Note that it is a common practice 
even for an experienced researcher to iterate this process. 


Task: Write up the outline of your mini research paper. Try to respond to each of the bullets in 
Module D Comprehensive Exercise Part 1 Task 1. 


Survey: Time spent between classes: 


// End 
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Mini Research 


* Questions? 
* Status? 
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Complexity Hierarchy 


Exp ae Crucial factor: Non-reusability of time 
(at least in a classic sense of time) 


i jones NPSPACE 
cw NP tee 


j= PSPACE 


pan 


v 
aes eee NLogSPACE 


LogSPACE 


Time Space 
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Effects of Nondeterminism 


Observation so far 
* L(TM) = L(NTM) 
* L(DFA) = L(NFA) 
+ L(DPDA) c L(PDA) |PDASareé nondeterministic! 
* PcNP 
* Probably P < NP (i.e., P # NP) 
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Unit D3 (cont'd) 


Today’s Questions 


1. Xvs. XSPACE 
* E.g., P vs. PSPACE, NP vs. NPSPACE 
2. XSPACE vs. NXSPACE 


* E.g., PSPACE vs. NPSPACE, LogSPACE 
vs. NLogSPACE 


3. Xvs. NLog(X)SPACE 
* E.g., Exp vs. NPSPACE, P vs. NLogSPACE 
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Group Exercise: Question 2 


* XSPACE vs. NXSPACE 


* E.g., PSPACE vs. NPSPACE, LogSPACE 
vs. NLogSPACE 


« Which would be correct? Why? 
A. XSPACE c NXSPACE 
B. XSPACE = NXSPACE 
C. XSPACE > NXSPACE 
D. None of the above 
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Savitch’s Theorem 


; ; SPACE (fin)): Space- 
¢ Given a function f(n) > n, bounded by O(/in)) 


NSPACE(f(n)) c SPACE (f7(n)) 
— Corollary: NPSPACE = PSPACE 
¢ Proof idea 


— Elimination of exponential factor: Simulate 
nondeterminism using a stack a la “iterative 
deepening” 


— Source of the square: Multiplication by the 


Stack height 
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Complexity Hierarchy 
Exp Fe Savitch’s Theorem 
‘ _-~7~ PSPACE < > NPSPACE 
. 
~-NP 


P 


v 


LogSPACE 


Time Space 
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X > NLog(X)SPACE 


* Exp > PSPACE idea 


« A TM using n space can have at most n x 20” 
different configurations (~ states, head 
position, tape symbols). 

* PSPACE would use Exp time. 

* P>NLogSPACE idea 

* MAZE is NLogSPACE-complete (i.e., can 
simulate any NLogSPACE problem). 

* Analogous to the above case 
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D3 Unit Summary 


* Time and space complexities have some 
connection. 


¢ The effect of nondeterminism differ in each 
case. 


* Applications of complexity theory 


— NPC and beyond => Use of approximation 
algorithms 


— Effect of limited resources (of different types) 
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Group Exercise: Question 3 


« Xvs. NLog(X)SPACE 
* E.g., Exp vs. (N)PSPACE, P vs. 
NLogSPACE 
« Which would be correct? Why? 
A. XcNLog(X)SPACE 
B. X= NLog(X)SPACE 
C. X>NLog(X)SPACE 
D. None of the above 
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Complexity Hierarchy 


~-..4  Savitch’s Theorem 
_-oe PSPACE <«--~-* NPSPACE 
ld 


_aww NLogSPACE 
LogSPACE 


Time Space 
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Unit D4: Overview 


* Analyze the effects of parallel processing 
on theoretical aspects 


« Explore examples of parallel processing 


* Preview Exercise D4 “Reading: Critical 
View about TMs” 


CSC460 D4 


Parallel Processing Overview: Theory of Computation 


* Classification 
— A few processors in a single computer . 
‘ Uninteractable? Eg@eus in Module D 
— Parallel machine | = 
— Distributed computing [multiple units/locations] interactable? 
* Questions Distributivity? slittige'stsal 
i A Decidable 
— Decide “undecidable” problems? real practicality camel 
— Move down Chomsky hierarchy? ee — ee 
; ; ; F t 
— Reduce time complexity (asymptotically)? Complexity] _Tactabie egulal 


impossibility |Computability 


/~ 
simplicity 
14 


- | 
practicality Languages 


| Automata 
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Review Review 


Computability Summary Extended Chomsky Hierarchy 


Non-TM-rec. TM-recognizable (RE) Grammar/TM 
Undecidable ee re 


Decidable (Recursive)*\---\----_ /Decider 
TM-recognizable 


Decidable Context-Sensitive----\--7---- CSG/LBA 


Context-Free} fo CFG/PDA 


\ ( Gooulery y RegExp/FSA 
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Review 
Complexity Summary Group Exercise 1 


“4 Savitch’s Theorem . Choose at least two different types of 
~~ PSPACE «----» NPSPACE parallel processing examples 
a 


. Could they decide “undecidable” 
problems? 


. Could they move down the Chomsky 


| hierarchy? 


. Could they reduce time complexity? 
LogSPACE 


Tim 
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Classification of Tasks 


Instruction 

— Parallelizable vs. inherently sequential 
Data 

— Disjoint vs. shared 
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Degree of Parallelism 


May consider different types ee 
* Constant, i.e., a fixed number k of - simp 


. MISD 
parallelism Daas 


* Function of the input size, n 
— Assumption: A large number of processors 
are available so that given any reasonable n, 
every c input units can be assigned a 
processor (for linear case, other functions 
possible). 


Massively parallel processors 
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RAM 


¢ Random Access Machine 


processor P 


pmemory cells |M,|M, 


m 


— Processing cycle: read-compute-write 
— Computation type: arithmetic 
— Cf. URM, which is more primitive 
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Classification of Parallelism 


SISD: Single Instruction Single Data 
SIMD: Single Instruction Multiple Data 
MISD: Multiple Instruction Single Data 
MIMD: Multiple Instruction Multiple Data 


Flynn [1966] 
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Group Exercise 2 


* Analyze whether the following cases of 
parallelism would affect: (a) computability, 
(6) complexity, (c) Chomsky hierarchy. 

Case 1: Constant factor 

Case 2: Function of the input size, 


— Assumption: A large number of processors are 
available so that given any reasonable n, every c 
input units can be assigned a processor (for linear 
case, other functions possible). 


csc460 D4 Modeling parallelism? — 55 


PRAM 


¢ Parallel Random Access Machine 


P processors 


p memory cells 


— Processing cycle: read-compute-write in sync 


— Processors run the same program, possibly 
different interpretation based on their index 


CSC460 D4 


Models of Parallel Processing 


* PRAM 

* Hypercube (2? processors for d 
dimensions) 

¢ Bounded Degree Network (at most d 
connections per processor) 
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Group Exercise 3 


Find the time complexity and processor 
complexity (number of processors) for 
the following problems: 

. Summation 

. Matrix multiplication ¢; =) a,b, for <i, j <n 

. Merge sort 7 
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Unit Summary 


* Parallel processing can affect complexity, 
and Chomsky hierarchy, but not 
computability. 

— Some (not all) common problems have 
exponential speed up with parallel processing. 

* The complexity of distributed computing in 
general is more difficult to analysis. 
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Example Problems 


Search 
Finding the max 
Summation : 
Matrix multiplication ¢; =).4n%; forlsi,j<n 
k=l 
Mergin 
ging Handling conflicts 


Merge sort [C: common, E: exclusive 
+ EREW [not realistic] 
+ CREW 
+ CRCW 
+ Common write 
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Class NC “Nick’s class” 


* Informally, efficiently parallelizable class of 
problems 

* Definition: Solvable on a PRAM in log time 
with polynomially-many processors 

* NC cLogSPACE c P 
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Summary Question 


¢ If parallel processing does not affect 
the computability results, what would 
do so? Speculate. 


* Questions/Comments/Suggestions 
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Name: 


Exercise D4, 4/15/05 


Reading: MacLennan 


While we have been studying the traditional, TM-based Theory of Computation as a useful tool 
for analyzing practical problems, we also encountered its limitations at various points. In order 
to broaden our view, we will read a paper by B.J. MacLennan, who is very critical about TM- 
based models of computation. 


Reference: MacLennan, B. J. 2003. Transcending Turing Computability. Minds and Machines 
13(1):3-22. [also see others in the course reference list; MacLennan’s web site: 
http://www.cs.utk.edu/~mclennan/] 


Task 1: Read the paper carefully. Try to relate the author’s argument with what you learned in 
this course. 


Task 2: Concisely write up your response to the following questions. In doing so, try to apply 
your analysis to your own mini research paper. 


What is the main research question of the paper? 

What is the significance of the research question? 

Does the paper respond to the research question well? 

Is the paper organized well? 

Summarize the author’s criticism of the traditional Theory of Computation in the order of 

importance (as you understand). 

6. Try to criticize the author’s position/arguments. If you cannot do this, explain why. 

7. How would the author’s criticism about the traditional Theory of Computation affect your 
analyses of your own problems and/or your choice of the mini research problems that have 
been done so far, focusing on computability and (time) complexity? 

8. [Optional] What would the author mean by “pitfalls of learning’? (in the middle of p. 19) 

9. [Optional] Discuss whether you would analyze yourself in terms of a Turing machine or a 
natural computation, referring to the arguments in the paper and relevant properties of yours. 

10. [Optional] The author does not directly discuss one subarea, formal languages/automata 
(except the TM). Speculate what the author would say about the Chomsky hierarchy. 

11. [Optional] Speculate on how to define the “power” of natural computation. (in the middle of 

p. 19) 


Sie NS 


Be prepared to discuss your response in class (no need to prepare for a formal presentation). 


Survey: Time spent between classes: 


Supplemental notes for the reading 
e = Effective (p. 4): Roughly, well-defined, step-by-step (e.g., algorithm = effective procedure that terminates) 
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e Calculus (p. 4): In logic, calculus refers to basically symbol manipulation (i.e., syntax, which is contrasted with 
semantics or meaning). 

e Phenomenological (p. 5): One of the main methods of the phenomenological investigation is to suspend 
presuppositions (Stewart and Mickunas, 1990). 

e Schema (sg.)/schemata (pl.) (p. 6): As an example, X + Y= Y+ X is arule schema, which can be instantiated 
with various values for X and Y. 

e Semantic vs. pragmatic effect (p. 8): While semantics normally refers to context-independent part of the 
meaning of a statement, pragmatics is about how to use symbols in context. 

e Generative grammar (p. 8): Basically the same as our use of “grammar” in this course. That is, a grammar is 
supposed to accept/generate strings. 

e Competence/performance (p. 8): These were introduced by Chomsky, and are widely used. Analogous (but not 
identical) contrasts have been noted by other people as well: e.g., langue/parole by Saussure, I-language/E- 
language by late Chomsky. 

e = Tarski (p. 8): The main figure who proposed the “truth-conditional” approach to interpretation of logical 
statements. 

e Lowenheim-Skolem Paradox (p. 10): Here is an example used in my section of Discrete Structures. 
Mathematicians try to specify the set of natural numbers using a set of statements in first-order logic. However, 
when they feel they are successful, the specification also specifies unintended structures as well. That is, all the 
structures of the form NZ*, where N represents natural numbers, Z represents integers, and ‘*’ represents the 
Kleene closure, i.e., any number of integers attached after natural numbers can still satisfy the specification. A 
result like this can only be obtained if we carefully relate the syntax (form) and the semantics (content) of a 
logical system (as done in model theory, part of formal logic). This suggests that there may be a lot of loose 
ends to formal specification. 

e =Satisfice (p. 12): “To accept a choice or judgment as one that is good enough, one that satisfies.” Herbert 
Simon is a Nobel laureate (Economics), who is also considered as a pioneer in AI. 

e Foreground/background (p. 14): Cf. people’s interpretation of the “vase-face illusion.” 

e Emergent phenomenon (p. 16): One of the main properties of a complex system. Such a phenomenon, which is 
often unpredictable, may emerge as a result of the interaction among (possibly relatively simple) components. 

e Pragmatics/semantics/syntax (p. 17): Pragmatics deals with how language (or other means) is used in context. 
Semantics primarily deals with a context-independent part of the meaning of a sentence. Syntax deals with the 
(recursive) structure of a sentence. 

e Nekker cube (p. 17): Best to see the image (do a web search). The switching of your perception can be 
analyzed in terms of “catastrophe theory” (a non-technical intro in, e.g., Gottman, et al. “Mathematics of 
Marriage”). This type of observation is often used in support of “holistic” position to cognition, cf. 
“reductionistic” or “analytical” position. 


// End 
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Unit D4 (cont’d) 
Example Problems 


Search 
Finding the max 
Summation : 
Matrix multiplication Ci =) a,b, forl<i,j<n 
k=l 
Mergin 
ging Handling conflicts 


Merge sort [C: common, E: exclusive 
+ EREW [not realistic] 
+ CREW 
* CRCW 
+ Common write 
CSC460 D5 + Priority write 


D4 Unit Summary 


* Parallel processing can affect complexity, 
and Chomsky hierarchy, but not 
computability. 

— Some (not all) common problems have 
exponential speed up with parallel processing. 


* The complexity of distributed computing in 
general is more difficult to analysis. 
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Question 


* Is the Theory of Computation relevant to 
all the real-world computational problems? 


¢ Why or why not? 
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Class NC “Nick’s class” 


Informally, efficiently parallelizable class of 
problems 

Definition: Solvable on a PRAM in log time 
with polynomially-many processors 

NC c LogSPACE c P 

Examples? 
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Unit D5: Overview 


¢ What’s wrong with TM or TM-based 
Theory of Computation? 
— Reading: MacLennan 
— More examples 
— Analyzing the source of the problem 

* Preview Exercise D5 “Reading: Super- 
Turing Thesis” 
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Group Exercise 1 


* Compare your exercises (reading 
MacLennan) 

* Prepare to present interesting points 
(possibly including similarities and 
differences) 
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Reading: MacLennan (1) Reading: MacLennan (2) 


What is the main research question of the paper? . [Optional] What would the author mean by “pitfalls of learning”? 
What is the significance of the research question? (in the middle of p. 19) 


Does the paper respond to the research question well? . [Optional] Discuss whether you would analyze yourself in terms of 
a Turing machine or a natural computation, referring to the 


arguments in the paper and relevant properties of yours. 
[Optional] The author does not directly discuss one subarea, 
formal languages/automata (except the TM). Speculate what the 
author would say about the Chomsky hierarchy. 

[Optional] Speculate on how to define the “power” of natural 
computation. (in the middle of p. 19) 


Is the paper organized well? 

Summarize the author's criticism of the traditional Theory of 
Computation in the order of importance (as you understand). 

Try to criticize the author's position/arguments. If you cannot do 
this, explain why. 

How would the author's criticism about the traditional Theory of 
Computation affect your analyses of your own problems and/or 
your choice of the mini research problems that have been done so 
far, focusing on computability and (time) complexity? 


CSC460 D5 CSC460 D5 


Where is This Community? Swarm Intelligence 


Social classes 

+ Within each colony, there is a strict division of labor. Some are 
warriors who defend their colony. The farmers collect leaves, and the 
domestic workers tend the mushroom beds and the young. 

Job description 

+ Farmers collect leaves. Well-worn trails are made as they travel to and 
from their colony. They are selective about the leaves they collect, and 
will often travel a long distance to find a certain plant. In this way, they 
spread out their consumption so that trees do not become stripped. 
In special chambers, domestic workers process the leaves into a pulp, 
makin: d of fertilizer n which mushrooms are grown. The’ . . 
Hale aba tie ee bed, ensuring that only ri kind Ms * Are ants intelligent? 


mushroom is grown, and they continually add additional leaves to - ’ : in? 
enrich the crop. * Is intelligence in the brain? 
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Where is This Community? Army Ants 


¢ The entire community consists of the near- 
blind, due to some genetic problem. 


* This community survived many years, 
even against the attacks of invaders. 


* Successful for millions of years. 


* How could they function? 
—E.g., how could they find food? 
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Swarm Intelligence Group Exercise 2 


¢ Each ant is not very intelligent. The Irishmen in the area (say, 100 of 

* Their colonies exhibit complexity them) like to go to the local pub when Irish 
approaching that of intelligence. music is played (Thursday evenings) only 

* The intelligence is in the colonies, possibly INE pub isnar crowded (52). UpiaG0 
including the environment. people). 


— Cf. the shortest path recorded in the 
environment (not known by each ant) How could they predict the pub is not crowded? 
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cf. Mass Stupidity 


El Farol Problem Can a TM design this? 


* Example attempts 
— Same as the last week 
— Average of the last four weeks 
— Trend in the last eight weeks 
+ If they can all predict, all of them would go 


and the pub will be crowded. False prediction 


* There is no absolute way to predict the 


crowdedness of the pub. ‘Implications? 
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Challenges in/around CS Group Exercise 3 


Operating System ¢ What would be the essential properties 
— Interprocess communication, I/O, memory that might make TM inappropriate for 
Software Engineering modeling the real-world (or computation in 
- Specification, verification Doomed to failure? general)? 
Al 
— Games, robotics, NLP, Cyc, 5th Generation Al 
* Cognitive Science 
— Theory of mind (cf. Fodor: mind as TM) 
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Observation Algorithm vs. Interaction 


The environment (context) changes over time. Algorithm Interaction 

— No control over adversaries. Closed-book exam Open-book exam 
— Future environment is unknown/unpredictable. Sales contracts Marriage contracts 
The environment may affect agents (including Rationalism Empiricism 

their thinking process) through interaction. Compositional Non-compositional 
— Nonmonotonic (e.g., belief change) Reductionistic Holistic 

TMs cannot handle the passage of time. Monotonic Non-monotonic 


TM learning is not impossible, but impractical. T™ must be something... 


Tentative conclusion: Interaction is non-TM- 


recognizable. 
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Elements of Modern Computing 


Interactive 
(external inputs) 


™ Distributed 
1s (separated in space) 


Parallel 
(simultaneous in time) [Wegner 1997] 
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Computer Science 


“goto” considered harmful: Structured 
programming 

“assignment” considered harmful: 
Functional programming 


“procedure” considered harmful: Logic 


programming, object-oriented programming 


* Too much of the TM-based “theory” 
considered harmful: model of interaction 
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Uncharted area 


real practicality 


We find parallel algorithms and distributed algorithms, 
but no interactive algorithms. 
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Paradigm Shift 


Astronomy: Ptolemaic > Copernican 
Physics: Classic — Relativistic 
Mathematics: 

— Hilbert’s program — Gédel’s incompleteness 
Linguistics 

— Transformational > Mildly Context-Sensitive 
Psychology 

— Behaviorism/cognitivism — Social/cultural 
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Extended Theory of Computation 


impossibility Computability 


unintergetapie? 
Interactable? iti 
zt ae Traditional area 
Distributivity? | Undecidable 
= 


Decidable 

me Ps 
Intractable | Context-free 
v2 “ey 


Complexit a Regular 


practicality Languages’ simplicity 


Automata 
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Unit Summary Summary Question 


¢ TMs are too limited as a model of “real” ¢ What would a more realistic model of 


ak Seiaue ae computation look like? Speculate. 
— Call for a more realistic model 


* The future of Computer Science may depend on * Questions/Comments/Suggestions 
a paradigm shift from the current TM-based 
theory to a new theory of interaction. 
— The current “Theory of Computation” hopefully 
provides useful information for this transformation. 
* It might even be the theory of “everything,” cf. 
Unified Theory in physics. But feasible? 
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Name: 


Exercise D5, 4/19/05 


Reading: van Leeuwen & Wiedermann 


After reading and discussing MacLennan, we may or may not agree with all of his points. 
However, one point we cannot overlook is that there is a growing number of researchers who 
take a position similar/related to MacLennan (see the course reference list for more information; 
Copeland [2003] is a detailed review). In order to gain additional insight into the area beyond 
TMs, we will read a paper by Jan van Leeuwen and Jiri Wiedermann, who propose a “‘super- 
Turing” thesis to capture the notion of computation at a higher level, potentially more applicable 
to modern computing. 


Reference: van Leeuwen, Jan and Wiedermann, Jiri. 2001. The Turing machine paradigm in 
contemporary computing. In Mathematics Unlimited - 2001 and Beyond, eds. B. Enquist and W. 
Schmid, 1139-1155. Springer. [the distributed copy, available on-line, is a draft version of (i.e., 
not necessarily identical to) the listed paper; van Leeuwen’s web site: 


http://www.cs.uu.nl/people/jan/] 


Task 1: Read the paper carefully. Try to relate the author’s argument with what you learned in 
this course. Note: You may skip the details of certain technical parts, e.g., the “advice” function. 


Task 2: Concisely write up your response to the following questions 

What is the main research question of the paper? 

What is the significance of the research question? 

Does the paper respond to the research question well? 

Is the paper organized well? 

Summarize the author’s arguments in the order of importance (as you understand). 

Were you convinced by Proposition 5 and its proof (p. 10)? 

What is the difference between the (traditional) Church-Turing thesis and the extended 

Church-Turing thesis (proposed in the paper)? What kind of impacts could the extension 

have? 

8. Try to criticize the author’s position/arguments. If you cannot do this, explain why. 

9. How would the author’s arguments affect your analyses of your own problems and/or your 
choice of the mini research problems that have been done so far, focusing on computability 
and (time) complexity? 


PLN Se OS 


Be prepared to discuss your response in class. 


Survey: Time spent between classes: 


Notes for the reading 

e = Oracle computation (p. 4): If a TM is given an oracle that decides some undecidable problem (set), then the TM 
could recognize a non-TM-recognizable language. Naturally, the increased power in this case comes from the 
power of such an oracle, which is beyond a TM. 

e Advice (p. 6): A limited version of oracle (because an oracle could be excessively powerful). Discussed more 
in detail in Section 3.2.1. 
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© Mapping ®: (2')* > (2')” (p. 7): A function whose input is an infinite sequence of k symbols on k ports and 
whose output is an infinite sequence of / symbols on / ports. As an example, suppose a machine with a 
keyboard and mouse as input device and display and printer as output device. Imagine that the keyboard deals 
with symbols from the set K; analogous for other devices: M for mouse, D for display, and P for printer. Then, 
the mapping/function would be as follows: ®: (K X M)rfor time 1) X (K X M) for time 2] X --- > (D X P)pfor time 1] X (D X 
P itor time 2] X +++ 

e Advice function (p. 8): The authors consider that oracle computation without limit is too powerful (p. 4). What 
they propose is to use a limited version of oracle called “advice.” Their “advice” takes the form of function: 
given a natural number, the function returns a string. The power of the function is basically “polynomial,” but 
include certain “randomization” possibilities, useful for reflecting unknown future. Here, we consider this 
function as a “reasonable” way to integrate mostly tractable but not completely predictable. Some additional 
notes are included below (indented). 

o Bounded by S() (p. 8): S(m) refers to “space bound” with respect to the input size (cf. T(n) for “time 
bound). Note that this bound applies to the F part of the class specification “C/F ” (see below). 

o Class C/F (p. 8): The main motivation for this definition is to introduce the class P/poly (see below). 
Skip the details. 

o Class EXPTIME (p. 8): This corresponds to the class Exp used in this course. 

o Class poly (p. 8): The class of polynomially (size-)bounded advice function. Recall that the space 
bound S(n) applies to F , and poly is one case of F . 

o Class P/poly (p. 8): Roughly, this class is a “non-uniform” (evolutionary, upgradable, etc.) analogue of 

P. It contains P (i.e., P < P/poly) as well as some undecidable sets (due to its upgradability, which is 

not fixed at the beginning). The comparison with NP is not yet known. The fact that P/poly contains 

RP and BPP (see below) suggests that P/poly captures some randomized behavior. 

Class NP/poly (p. 8): Ignore this class. 

o Class RP [Randomized Polynomial Time] (p. 8): A language is in RP if there is a boolean verifier 
computable in deterministic polynomial time (or finding a boolean satisfiability answer with a NTM) 
such that the input is accepted at least 1/2 of the time. 

o Class BPP [Bounded-Error Probabilistic Polynomial Time] (p. 8): Analogous (or more confident 
version) to RP, except that the input must be accepted at least 3/4 of the time (the value could be 
others, e.g., 2/3, as long as it is strictly greater than 1/2, as they can approach | by repeated 
multiplications). 

¢ automata (p. 9): Variants of FSAs that accept infinite strings (not just potentially infinite as in the case of 
FSAs). 

e Non-recursive (p. 10): Beyond recursive, i.e., r.e. (TM-recognizable) or non-r.e. (non-TM-recognizable). 

¢ (M) (p. 10): A standard way of indicating that this is a representation of a TM. We could consider this as the 
TM’s formal definition, as discussed in class. 

e Proof of Proposition 5 (p. 10): The basic idea is that the advice function identifies the maximum number of 
steps for a TM whose representation is of some length. Although the advice function adds power to the TM, it 
is still within the bound. So, it is not considered unreasonably powerful. However, we know that such a 
function is not decidable by a standard TM. 

e Theorem 6 (p. 11): Ignore it. 

e S=(y) (p. 11): Here the site machine is defined (as a 1-tuple) in terms of the function y from time to a 
hardware/software description (p. 7). 

e Instantaneous description (p. 11): A snapshot of a machine. The instantaneous description of a TM (or similar 
machine) includes the current state, head position, and the tape symbols. Note that if m tape positions are used, 
there would be at most |I|" symbol possibilities, where Tis the set of tape symbols; this suggests that it would 
have taken at most exponential time to have used the space. 

e Infinite circuit families (p. 16): Ignore this. 

e BSS model (p. 16): After Blum, Shub, and Smale. The problem of deciding general existential formulas in the 
first-order theory over the reals, which is a real analogue of NP-complete. 

e Neuroidal networks (p. 16): A specific version of a neural networks, proposed by a computational learning 
expert Valiant. 


O° 


// End 
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Unit D6: Overview 


¢ Explore models of super-Turing 
computability 

— Reading: van Leeuwen & Wiedermann 
— Other models 


Group Exercise 1 


* Compare your exercises (reading van 
Leeuwen & Wiedermann) 

* Prepare to present interesting points 
(possibly including similarities and 


* Summaries: Unit, Course differences) 


* Exercise D6 (Module D Comprehensive) 
“Mini Research” 
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Reading: van Leeuwen & Wiedermann Sample Problems 


What is the main research question of the paper? 
What is the significance of the research question? 


Driving from one place to another 
Does the paper respond to the research question well? “Real” web prog ram mi ng (arach nidan) 
Is the paper organized well? 


Summarize the author's arguments in the order of importance (as you Bird flocki ng 
understand). 7 . 
Psycho-social behaviors 
What is the difference between the (traditional) Church-Turing thesis and 
the extended Church-Turing thesis (proposed in the paper)? What kind 
“ a” . ‘ 
Try to criticize the author's position/arguments. If you cannot do this, Gaia hypothesis (the enti re earth as an 
explain why. organ ism) 
problems and/or your choice of the mini research problems that have The un iverse 
been done so far, focusing on computability and (time) complexity? 


Were you convinced by Proposition 5 and its proof (p. 10)? 

—E.g., child-parent attachment, language 
of impacts could the extension have? 
How would the author's arguments affect your analyses of your own 


Super-Turing: Requirements 


Super-Turing: Assumptions 


¢ Interactivity: Can accept input and deliver 
output in the middle of computation 


Extension: The TM-based “theory” is to be 

a special case of the theory of interaction. 

* Robustness: Can deal with fuzzy, Multiple levels of representation (cf. 
unknown, unpredictable environments Church-Turing Thesis) Why this many? 
(inputs) — Machine models (cf. TMs, URMs, RAMs) 

* Adaptability: Can change according to the — Mathematical/functional models (cf. recursive 
environment (i.e., learning) functions, -calculus) 

* Ockham’s Razor: Choose the simplest if 
multiple options are available 
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Super™ Models 


Site machines (concrete machine model) [van Leeuwen & 
Wiedermann] 


TMs with advice function (abstract machine model) [van 
Leeuwen & Wiedermann] 


Interaction machines (persistent TMs) (abstract 
machine model) [Wegner & Goldin] 


yet another machine model (intentionally kept untold at 
this point) 


T-CalCulus (functional model) [Milner and others] 
Neural models [e.g., Siegelmann] 


Quantum/relativistic computing cf. on-line references: 
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Variations 


* General: MIM (Multiple-stream IM) 
— Model of distributed computing 


* Special: SIM (Single-stream or Sequential 
IM) 


— Model of 2-agent interaction 
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Properties of SIMs 


SIM states depend on the content of the 
working tape [no bound]. 


Infinitely many states (cf. finite TM states) 


Uncountably many SIM’s (cf. countably many 
TMs) 


Can handle uncountable sets (cf. Tv can 
handle countable sets, i.e., uncountable ~ unsolvable) 
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rs 


Interaction Machine (IM) 


Note: TM computation = Single session 
IM computation = Multiple sessions 


Interactivity: Realized as repeated TM 
sessions where information can be stored 
on a working tape 


Multiple input streams on multiple tapes 
An extension of a multiple-tape TM 
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Single-stream IM (SIM) 


* Single input stream on a single tape 
* Work tape will be preserved (persistent). 


¢ Behavior: set of I/O streams (cf. TM behavior: 
set of I/O pairs) 


Example: Answering machine 
(input, work) => (output, work) 
* (record Y, X) => (ok, XY) 

* (playback, X) => (X, X) 

* (erase, X) => (done, €) 
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Properties of MIMs 


* No single observer (on a single stream) 
has complete information. 


— Cf. A SIM has complete information (about the 
interaction between the two parties). 
— Analogy: “Blind men and an elephant” 
¢ Multiple streams cannot be serialized. 

Example: Delegation 
+ Stream 1: (main-in, main-out) 
* Stream 2: (sub-out, sub-in) 
* Coordination: (main-in, sub-out, sub-in, main-out) 


CSC460 D6 


Comparison Analysis of IM 


TM: 1-agent systems <= countable sets 

— Traditional Church-Turing Thesis for 
algorithmic computability (TM, A-calculus) 

SIM: 2-agent systems — uncountable sets 

— Extended Church-Turing Thesis for sequential 


interaction (SIM, calculus?) “TM as amnesic SIM 
MIM: n-agent systems = ? 


— Ultimate Church-Turing Thesis for general 
interaction (MIM, calculus?) 
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* Interactivity 
¢ Robustness 
* Adaptability 
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Functional Model A-calculus 


¢ TM © A-calculus 
- SIMs? 
-MIMe? 


Formal representation of functions 
¢ Abstraction: Ax.exp [define a function] 
* Application: exp, exp, [give an argument] 


* B-reduction: (Ax.exp) y [use a function] 


— The result is exp where the instances of x is 
replaced with y. 


—E.g., (Ax,f(fa))) 0 = AKO) 
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A-calculus Examples 


Example Scenario 


* Identity function: Ax.x 
¢ Church numeral 
+ Representation of a natural number n: A£Ax.f n(x) = 
+ O=AFAX.x [Note: AFAx.f0(x)] io. 
= ete Station Station 
* 1= Succ 0 =AFAXx.(f (((ALAX.X) f) X)) = ALAX.(F(AX.X x) 2 
= AFAX.(f xX) [Note: AfAx.f 1(x)] 
* Conditional Gaal 
* T=Axiy.x [i.e., picking the first argument] 
+ F=Axay.y [i-e., picking the second argument] 
* BMN (if B then M else N) where B is T or F 
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m-Calculus 


Changes to A-calculus 


* Output mechanism 
— Cf. B-reduction: (Ax.exp) y as an input mechanism 


« Parallel composition of processes 
— Commutative, associative 
* Meaning as state change 


* Referencing (by name) 
— Cf. pointers in modern programming languages 
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Analysis of z-Calculus 


* Interactivity 
* Robustness 
* Adaptability 
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Extended Theory of Computation 


impossibility (Computability 
Uncharted area ii 
Uninteractable? 


Traditional area 
Distributivity? | Undecidable 
2S 

Decidable 

ri ~N 
Intractable  Context-free 
a fay 

Regular 


real practicality 


Complexity] Tractable 
2 


practicality Languages simplicity 
Automata 
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m-Calculus Example 


def 


CAR(talk, switch) = ot [talk, word |.CAR(talk, switch) 
VAlswitch, ch, |r |switch, ch, lcar (ch, sch, ) 


al talk 
ao 


Station, Station, 


Control 


def 
NETWORK = (CAR(talk,, switch, )| STATION, |...| CONTROL) 


20 
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Super-Turing Thesis 


* Would all models of interaction converge? 


* Would it be the next level of 
computation/interaction/emergence? 
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Unit Summary [critical view] 


¢ TM-based theory 
— Stimulus-response without thinking 
— Vertebrate with a disabled brain 
Cf. “chicken without a head” 
* Theory must evolve into a more 
sophisticated beast. 
— Modeling full-fledged interaction 
* Some building blocks are emerging. 
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Preview 


Module D Evaluation Workshop Course Summary 


* Mini research presentations ¢ Understanding the “Theory” = Computer 
— Know the time limit; Think big; Be creative scientist (good at solving algorithmic problems) 
* Peer discussions ¢ Understanding the limitations of the 


“Theory” = Philosopher (good at solving “real” 
problems) 


* Change in your brain? 


¢ Reflection and self-evaluation 


Sin minas, no hay tango. 
Sin teoria, no hay ciencia. 
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Exercise D6 (Module D Comprehensive Exercise), 4/22/05 


Note: This exercise is announced in advance so that we can think about it through this module. 
The due date of this exercise is the date of Module D Evaluation Workshop. 


Part 1: Mini Research 


One of the main goals of this course is to be able to see real-world problems in a principled way 
assisted by the concepts and tools in the Theory of Computation. To practice making such a 
connection between problems (concrete) and the Theory (abstract), you will spend part of your 
out-of-class time to conduct mini research. You will identify your own research question which 
is related to a practical problem of your interest, and respond to it using the skills gained in this 
(and possibly other) courses. Your research question can be related to your initial problem 
and/or earlier mini research questions, or can be an entirely new one. If you choose a related 
question, avoid repeating what you have already found and/or written; that is, if you continue on 
an earlier topic, you must have something new (and must explicitly state what is new). If you 
want to work on a borderline/questionable case, consult the instructor. Some parts of module D 
take-home exercises will guide you through the process of completing this comprehensive 
exercise: e.g., Exercise D2 asks for the choice of your research question and Exercise D3 asks 
for an outline of the paper. 


Task 1: Write a mini research paper. In your paper, address the following points: 

e Clearly identify your research question early in the paper (if necessary, provide some 
background information to introduce the question) 

e Clearly identify the cost/significance of the research question (this can be done by connecting 
your research question with a practical problem) 

e Organize your paper so that the paper responds to the research question applying the 
concepts/techniques discussed in this course, at a level reasonable for the available time and 
your capacity 

o It would be ideal if you can refer to all three subareas of the traditional Theory of 
Computation. If this is not possible, try to include as many Theory topics as possible. 

o While it would be great if you can offer a “solution” to your research question, you 
should be content if you describe your attempt to solve it. 

e Conclude the paper with a punch-line statement of your response (and if you were unable to 
offer a solution, state the information necessary to solve or complete your research) 

e Consider to have multiple sections (Introduction, middle section(s), and Conclusion). 
However, there are no other formatting or length requirements. 


Note: Recall that paper writing (as well as coding) can be seen as a context-free activity (if you 
cannot remember, make sure to understand this by discussing with other students or the 
instructor). When you organize your paper, try to apply this concept. 


Task 2: Prepare for a five-minute, informal presentation (to take place during Module D 


Evaluation Workshop). You can (but not required to) use PowerPoint slides (no more than 
several slides) if the file is accessible without logging into your Novell account (e.g., stored in 
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your public HTML folder on a floppy diskette). Attach to this exercise the printout of your 
presentation materials, if any (preferably, in the format with multiple slides per page). 


List of sample research questions from Exercise A6/B0 


Can organizational dynamics be modeled as an algorithm? 

Can evolution be modeled as an algorithm? 

Can ecology be modeled as an algorithm? 

Can human development be modeled as a computer? 

Can our minds be modeled as a computer? 

Can vision be modeled as an algorithm? 

Can learning be modeled as an algorithm? 

What would be the minimal mechanism to process human language? 
Can the entire situation of an arbitrary game be modeled as an algorithm? 


. Would “perfect” user modeling, e.g., for web search, be possible? 

. Would “perfect” computer security be possible? 

. Can the entire process of software engineering be modeled computationally? 

. Can computer networks be modeled as a single computer? 

. Could biology be reduced to physics? 

. Could some computer generate real (not pseudo) random numbers? 

. Would it be possible to decide whether the given numbers are random? 

. Would randomization affect computability and/or complexity? 

. Would parallelism affect computability and/or complexity? 

. Would artificial neural network be more powerful than TMs? 

. Would cellular automata be more powerful than TMs? 

. Would the use of analog (or fuzzy) values affect computability? 

. Would relativistic, quantum, or some other modern-physics-based computation surpass TMs? 
. Can all the cases of on-line algorithms be simulated by off-line computation? [On-line algorithms would 


obtain inputs as the time progress. Off-line computation would provide all the possibilities as input at once. Cf. 
function-to-relation conversion used to fold the output within the input] 


. Would oracle computing affect computability? [Oracle computing: A TM with the capability to ask questions 


to another mechanism] 


. Would persistent TM be able to compute more than the standard TM? [Persistent TM: Multiple sessions of 


TM operation with some memory between them] 


. Would accelerating a TM give more power? 

. Would slight error tolerance affect any aspect of the Theory of Computation? 
. What would be the ability of a finite automaton with a queue? 

. What would be the effect of “constant” (as in complexity analysis) in practice? 
. Can any mathematical function be represented computationally? 

. What exactly are power sets doing to the Theory of Computation? 

. Is what you can do in logic the same as what you can do with computation? 

. If you have a research question of your own, please consult the instructor first. 


Part 2: Evaluation Form and Supporting Notes 


Review the evaluation procedure. Then, complete your evaluation form and supporting notes. 
Bring them to the evaluation workshop (hard copy). Print them well in advance so that you can 
avoid potential problems, e.g., not being able to print just before the evaluation. 


Survey: Time spent between after Unit D6 before the evaluation workshop: __ 


// End 
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Practicum Evaluation, 4/27/05 


In this course, critical analysis has been one of the main criteria associated with a learning goal. 
Since you have been through various topics in the theory of computation, you are in a good 
position to review and give constructive feedback to your peers. So, take this opportunity 
seriously and try your best to contribute the student body. Although this event takes place after 
the final evaluation workshop, it still is a part of the course evaluation. The instructor will check 
the criterion only after receiving this sheet accompanied by your evaluation forms and confirmed 
that you demonstrated your critical analysis skills. As in other exercises, no grades will be 
assigned. The instructor may note his comments on your evaluation, if necessary. 


Here is the procedure: 


1. Identify the presentations you are assigned to evaluate. The information is available on-line 
(linked from the CS home page) and posted everywhere. 

2. Obtain practicum feedback forms and fill in one form per presenter. Unless otherwise 
required (e.g., by the instructors of other courses), keep your forms anonymous. 

3. When you are done, hand in your forms to the instructor attached to this form. DO NOT 
deposit the forms in any of the appraisal boxes. 

e = The instructor will make photocopies of your forms, and will forward the original 
forms to another instructor or deposit in an appraisal box, depending on the 
information you supply below. 

4. If you are required to evaluate also by other instructors, please list the names of the 
instructors below. If they have specific requirements, by all means satisfy them. 


Instructors who also require evaluation: 


// End 
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e See the supplemental notes to Unit D3 on QBF and time-space tradeoff here. [4/13/05] 
e Asample response to Ex D1 Part 1 (big-O notation via the graphing tool) is available here. [4/8/05] 
e Link to an interview with Dean Kamen, the inventor of Segway, on NPR Fresh Air. He emphasized the 
importance of crossing between concrete and abstract ideas. [4/7/05] 
e See the supplemental notes to Unit C6 on "problems" and "languages" here. [4/5/05] 
e There are a few additional materials: [3/7/05] 
°o Compilation of Ex B5 heuristics 
o Additional notes on Church-Turing thesis and Rice's theorem 
o Extra problems on computability (optional) 


e My response to your summary questions in Unit B4 is available here. [2/25/05] 

e An example of binary adder is added here. [2/7/05] 

e Sample problems from Spring 2003 are availalbe here and also among the top bar selections. [1/28/05] 

e A preliminary version of the references for the course is available here. This file will be updated in the 
future. [1/25/05] 

e Welcome to CSC460 course web site! A preliminary version of the syllabus is up now. The final version and 
more components will be added as they become available. You are welcome to browse the web pages. 
You can at least see a little more details about the schedule, evaluation, etc. There are certain aspects of 
this course that are different from many other CS courses. Please find out whether you are willing to do 
this. By the way, there is nothing you need to do until the first day of class. Furthermore, you'd better wait to 
buy the textbook, because | will comment on how we will use it. [12/22/04] 
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CSC460 Theory of Computation (Spring 2005) 
Reflective Essay 

Nobo Komagata 

May 4, 2005 


For the past few years, I have been sharing my own reflective essay with students at the end of 
each course. Here is one for this course; it’s a little long. Note that the links in this document 
are clickable. 


Grades 

Your course grades have been reported to the College; both your Module D and course grades 
are also available on SOCS (https://socs.tcnj.edu/). I started these notes with “grades” not 
because it is most important (although some student might think that way) but because it is the 
least interesting part for me. In my opinion, the letter grade you receive for this course (or 
possibly other courses as well) means almost nothing. Throughout the semester, we have been 
evaluating ourselves with a substantial amount of information, which is obviously not reducible 
to a single letter. So, I hope you know your “real” assessment. I think you all achieved the 
learning goals as a result of good deal of effort. 


Practicum Evaluation 

First, I would like to thank you for participating the practicum presentations as evaluators. The 
main reason for this requirement was to promote practicum participation. This stems from some 
faculty members’ experience until a few years ago, when practicum presentations were taking 
place often with a very small number of audience. Probably, participation as requirement is not a 
solution. If presentations are sufficiently interesting and informative, there will be audience. In 
Unit B7, which was optional and was attended by four students, I asked them whether the 
College would function if there are no requirements. The response was negative. But there are 
some aspects of life which are not required but done regularly. Are people required to watch 
TV, eat junk food, marry, etc.? My personal preference is to live in a society where people work 
not because they are required but for other (hopefully good) reasons. 


Mini research 

The submitted papers are now available on-line 
(http://www.tcnj.edu/~komagata/csc460/05s/students/). It was quite interesting to see your 
contributions to the final evaluation workshop, despite the limited time you had for the project. I 
felt that the workshop appeared as some sort of collective thought or intelligence (cf. global 
consciousness). It was a kind of forum where the participants were able to share ideas relevant 
to the course topics, while no single participant was able to predict the progress of the workshop. 
So, to me, it was a good example of complex system 
(http://en.wikipedia.org/wiki/Complex_system), which I will occasionally touch on below . 


Instead of writing comments on each presentation, let me write a single prose to address the 
presentation topics. Although the topics of this course is by all means the Theory of 
Computation, we talked a lot about other things. As we discussed frequently, unless we can 
relate the course topics to our lives, it is hardly possible to get interested. I think you all did a 
good job on this point. Sometime last year, you all decided to take this elective course. Was it a 


result of “determinism” (1.e., everything is pre-determined) or did you apply your “free will?” 
Did you do things when they are optional? Were your decisions affected by any of the 
following: (1) massive parallel processing with large amounts of concurrent data, (2) ability to 
adapt, and/or (3) some element of randomness (Scott)? While most currently used programs 
would not satisfy all these conditions, if there is one with these three properties, would it behave 
like us, exercising “free will?” An interactive TM with advice (van Leeuwen & Wiedermann) 
can satisfy these; advice function can contain some randomness. In a sense, nondeterminism 
may be broader than free will. Have you seen a movie called “Sliding Doors?” It is a charming 
film, which illustrates a potentially large effects of a very small difference (cf. “butterfly effect” 
in chaos theory). This is also related to a source of the failure of symbolic AI, 1.e., “ramification 
problem.” A related but slightly different property is “predictability.” If everything is 
predictable, we could collect all the future inputs and give them to a TM all at once (“off-line 
computing’) instead of feeding the information as it becomes available (“on-line computing’’). 
Off-line computing is an active area of research. However, if the advice function of a site 
machine involve unpredictability, it may not be able to realize everything off-line. 


Regardless of our philosophical position, it is certain that our brains work hard to bring about 
various changes in our bodies. For example, while strolling a shopping mall, we may be using 
our memory in a way analogous to a PDA (Marisa). However, in general, memory appears 
much more flexible. Even remembering a story would be beyond context-free because we can in 
general recall any part of the story (cf. Marisa). Now, recall that any (standard) TM-based 
model is “amnesic” in the sense that the result of a single session of computation is not retained. 
Many super™ models overcome this deficiency by the use of some sort of persistent storage. 
However, even this modification may not be accounting for human memory. For example, 
human memory is said to be constructed rather than recalled (Engel). What kind of model would 
explain cases such as anterograde amnesia (Mike)? Maybe, we will be able to signal a 
problematic area of the brain from a PDA through MemoryGate (cf. Jared). 


In practice, though, we are far from such a goal. In general, it is impossible to “perfectly” model 
most of natural phenomena (cf. Jon). Natural computation involving “continuity” would face 
uncountability, beyond TMs (Jon). But even countably infinite inputs cannot be placed on the 
tape of a TM. It would also be impossible to accomplish “perfect” computer security (Megan). 
However, the notion of being “perfect” is difficult to nail down. While a computer scientist 
might acknowledge “perfect” computer security if someone proves P # NP (Megan), one might 
say that a “perfectly” secure system must be impossible to crack. Being in NP simply suggests 
that it take some time; it certainly accepts that it would be possible to crack (with unrealistically 
long time). Well, I feel that the demand for remembering all the passwords is way over my 
computability. In fact, I have a password-protected file that stores passwords, which are again 
encrypted in my own mnemonic. Honestly, I don’t think this is the future of computer security. 
In a sense, algorithm-based cryptography is an object-level approach to computer security; in 
many cases, one needs to deal with security issues at a meta-level. Often we get stuck at the 
object level and are not seeing the same problem from a meta-level; this situation can be 
observed in various cases including competitions/games. In a sense, computer security can be 
characterized as an evolutionary game between computer security developers and hackers. A 
new ideas in this game might be more evolutionary than algorithmic. 


As a tool to capture some aspect of evolution, cellular automata (CAs) are becoming a common 
tool. While it has been shown that CAs can simulate TMs (Wolfram via Mark), the other 
direction is trickier (cf. Mark). Wolfram shows that some segment of CA behavior can be 
simulated by a TM. However, since CAs would continue indefinitely, unless all the ever- 
growing patterns can be classified based on a TM computation, it would be premature to say that 
TMs can simulate CAs. This may be the case, because it has been shown that the bulk of CAs 
can be reduced to a set of deterministic patterns (Israel & Goldenfeld). But the most striking 
claim of Wolfram seems to be “Principle of Computational Equivalence,” roughly “almost all 
processes that are not obviously simple can be viewed as computations of equivalent 
sophistication.” To me, this suggests some kind of computation beyond TM. However, it is also 
true that CAs appears more limited in terms of (external) interaction compared to, say, site 
machine. 


In addition to memory, other types of cognition were brought up. For example, while we 
humans process vision in real time, it still would be non-TM-recognizable partly due to the 
involvement of learning (John). Another aspect of cognition, common sense knowledge is the 
focus of a long-standing project called CYC (Jared). This project seems to be following the 
failed track of the Fifth Generation AI, possibly due to its heavy reliance on logic. While first- 
order logic (FOL) is extremely useful on various fronts, it has its own limitations. When Hilbert 
proposed that all of mathematics can be formalized in FOL (“Hilbert’s program”) and Russell 
tried to complete the program, Gédel showed that a consistent logic that can count cannot prove 
all true sentences (“Goédel’s Incompleteness Theorem’’) (Greg). As mentioned in one of the 
supplemental notes, this result is directly related to the undecidability of FOL. As for 
complexity, universal quantifiers are bottleneck (Greg), for proving the sentence, corresponding 
to the burden on the falsifier. But due to the “duality” of the universal and existential quantifiers, 
disproving a sentence calls for analogous burden on the verifier. So, to be able to say whether a 
sentence is true or false, both types of quantifier contribute the computation equivalently, leading 
to exponential growth. Pointing out various problems with FOL, Hintikka proposes a variant of 
FOL which is more tightly controlled by game-theoretic semantics (Hintikka). Although this 
idea has not been taken up seriously by traditional mathematicians, it might be wise to listen to 
such a voice; we might actually observe a paradigm shift in logic, parallel to the Theory of 
Computation. Note that the syntactic analysis of FOL is a quite different aspect. The backbone 
of FOL sentences can be “parsed” with a PDA (Greg). However, just like modern programming 
languages, variable coreference requires more than that. 


While cognition is more actively studied by psychologists, emotion may hold a key to mind- 
computer analogy. It is common to think that emotion cannot be “computable” (Marisa). I am 
in general sympathetic to MacLennan’s arguments and tend to emphasize the limitations of 
algorithmic computation. But here, I would like to point out some algorithmic aspects of 
emotion. Have you ever mistaken a coarsely rolled rope on the ground as a snake and jumped 
with fear? We can obviously recognize a snake, but it takes time, suggesting that vision is not so 
cheap (John). To avoid potential danger involving a snake, our brains are also equipped with a 
sort of “short” circuit of emotion that induce a quick action without detailed visual processing 
(LeDoux). There are many other aspects of emotion that can be reduced neurobiology. For 
example, it has been studied how infants process visual information and that lead to emotional 
change in their bodies (e.g., Schore). We also know that an appropriate amount of emotion (e.g., 


frustration, comfort from sharing ideas, surprise from discovery) is essential for learning (note 
that excessive emotion would lead to traumatic amnesia). If all sorts of emotions can be 
analyzed and synthesized in the form of robot, it might be possible to generate what we call 
emotion in a non-organic form. Then, there might be a pathway from a PDA to an anterograde 
amnesic (Mike) ora TM. But until such a day comes, if at all, we have music to appeal to our 
emotion directly. 


As the Church-Turing thesis defines the informal notion of algorithmic computation, it would be 
possible to define the informal notion of enjoyable music (Eric). But unlike “algorithms,” which 
are more objective, musical enjoyment is more subjective. Then, would it be really recursive (cf. 
Eric)? For example, our tastes change over time (after listening to a Cuban pianist, Rubén 
Gonzalez, I started to feel some other pianists are no longer as good) and are often unpredictable 
(there are too many musical “strings” which I have never encountered). We might want to 
respond to such a question by focusing on the domain where TM can decide, or using a super’™ 
model instead. 


This course 

As a part of my reflection, I would like to write a little bit about this course from my view point. 
A preliminary version of this course was run two years ago as a special topic course. At that 
time, I was not focusing on complex systems as much as today. As a result, I emphasized the 
significance of the Theory more than its limitations. That is, I was trying harder to “sell” the 
Theory. It was not as successful as | expected. During that semester, I was asked to teach 
IMM220 “Principles of Interactivity.” The more I thought about interactivity, the more I felt 
difficult to sell the traditional Theory. At the end of that semester, I prepared an earlier version 
of super-Turing lectures, mainly based on materials by Kennedy & Eberhart, Wegner, Goldin, 
and Milner; I was not aware of MacLennan and van Leeuwen & Wiedermann then. This 
semester, I approached this course more critically, toning down the significance of the Theory 
and discussed a lot more about the limitations, as you are well aware of. I am glad to have done 
this. While I used the traditional Theory a lot during my graduate work (e.g., decidability 
analysis of pragmatic process, grammars/automata to analyze/process natural language syntax, 
complexity analysis of parsers), there are so many areas/phenomena that escape the traditional 
Theory. You know the story. 


Although I re-used many slides from the previous run, I revised the course substantially: new 
learning goals, new exercise problems (and exercise pattern) including new reading, new 
evaluation procedure, new in-class exercise discussion pattern (designated pairs), etc. Again, 
I’m glad that I did these. I tried to write more about “why” we do certain things (1.e., 
significance), e.g., at the beginning of each exercise. I do frequent exercises because I know 
from the education research literature and my own experience that shorter frequent exercises 
with timely feedback are more effective than longer infrequent exercises. At the same time, I let 
you decide on the level of detail so that you can adjust the amount time you use for each 
exercise. I also tried to place the materials in context (not as successful as I hoped, though). I 
am usually frustrated with textbooks on this point, e.g., Section 1.1 of our text, which is 
supposed to explain why one should study automata. On another front, reading technical papers 
in this area tends to be too demanding for undergraduate students. I didn’t do it two years ago. 
But this time, I felt that the two papers are indeed readable and beneficial for promoting your 


critical attitude. I also tried to promote collaborating atmosphere; you did a good job on this. 
Although everyone of us has some competitiveness, excessive competitiveness can be very 
harmful (Rosenau). 


Over the past few years, I have been revising the evaluation procedure, partly based on the ideas 
in the education literature (e.g., Huber & Freed, Fink). Whatever I do, there always are students 
who don’t like it. But I cannot stop experimenting with new approaches/materials. My wife 
(also a college teacher) and I often wonder why so many college teachers (who did a PhD) don’t 
apply their research attitude to teaching. Not every attempt would be successful, but no attempt 
must be self-limiting. In the end, I am glad that I did this course. 


One other thing, which is personal, is that I got a Da.D. on March 14. I tried to minimize certain 
adverse effects on this course, but I have to admit that it was the single most challenging event I 
experienced in my life (even more than a Ph.D.). But I hope you enjoyed some baby examples. 
As I mentioned, child-parent interaction is highly complex (Schore). Actually, there is a book on 
this topic written from a complex systems point of view (Siegel), which was a textbook for my 
section of First Year Seminar last semester (http://www.tcnj.edu/~komagata/fsp111/04f/). In 
case you plan to become a parent, I highly recommend this book; you may even find something 
new about the relationship with your parents. 


Thank you for taking this course (with your free will) and contributing to the class interactively. 
I’d be happy if you learned something in this course as a result of your own motivation (not 
because you were required to do so). That way, you will continue to learn. Regardless of your 
career choice, I wish you the best! Please let me know what happens next in your life. 


Additional References (only those not in the course on-line references) 

[L] indicates that the resource is available in the TCNJ library 

e ~=Engel, Susan. 1999. Context is everything: the nature of memory. W.H. Freeman. [L] 

e = Fink, L. Dee. 2003. Creating Significant Learning Experiences: An Integrated Approach to Designing College 
Courses. Jossey-Bass. 

e = Huba, Mary E. and Freed, Jann E. 2000. Learner-centered assessment on college campuses: shifting the focus 
from teaching to learning. Allyn and Bacon. [L] 

e Kennedy, James F., Eberhart, Russell C., and Shi, Yuhui. 2001. Swarm intelligence. Morgan Kaufmann 
Publishers. [L] 

e LeDoux, Joseph. 1999. Emotion, Memory, and the Brain. In the Scientific American book of the brain. Lyons 
Press. 

e  Rosenau, Pauline Vaillancourt. 2003. The competition paradigm: America’s romance with conflict, contest, and 
commerce. Rowman & Littlefield Publishers. 

e Schore, Allan N. 1994. Affect regulation and the origin of the self: the neurobiology of emotional development. 
Lawrence Erlbaum Associates. [L] 

e §=Siegel, Daniel J. 1999. The developing mind: toward a neurobiology of interpersonal experience. Guilford 
Press. [L] 
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The Notion of “Theory” in Formal Logic 
Nobo Komagata, January 14, 2005 


While the notion of “theory” is used in all sorts of context, we rarely agree on the term completely. As a background 
discussion to understand the notion better, these notes explores the idea developed in formal logic. The materials are 
taken from standard logic textbooks [Enderton, 2001; Ebbinghaus et al., 1984] and a textbook on program specification 
Loeckx et al. [1996]. 


1 Logic 


The presentation of logic in this subsection is intentionally left rather vague to abstract away from the details. A formal 
presentation based on first-order logic and many-sorted signature are found in the texts. 
Informally, a logic is a specification of a language and its meaning. We first introduce the components of logic. 


Definition 1 (Language) A language is a specification of the use of symbols. 


Depending on further specification, these symbols are combined into a term, a formula, or a sentence. For example, 
a language of first-order logic may have a term wife (Socrates), a formula man (x) > mortal (x), and a sentence 
Vx [man (x) > mortal (x)]. A language of equational logic may have a formula x + 1 = 1+ x. Although it is not 
technically correct (e.g., Enderton [2001]), we call the expressions we deal in these notes (in correspondence to 
“statement”) as “formula.”! 


Definition 2 (Interpretation) Interpretation is a function (mapping) that specifies the meaning of symbols built in the 
logic under consideration. 


For example, a logic of arithmetic may have a special constant ‘0’, whose meaning is fixed in the logic (e.g., the integer 
‘0’ as we understand). Many logics also have logical connectives such as ‘A’ (and), whose interpretation may roughly 
correspond to that of natural language “and.”” 


Definition 3 (Structure) A structure (for a logic) is a function (mapping) that defines the meaning of symbols not 
defined by the interpretation. 


By convention, for each sort (type) of elements, a structure assigns the universe of individuals corresponding to that 
sort. For each operation (in first-order logic, function or relation), a structure assigns its meaning in terms of the in- 
volved individuals. For example, a first-order logic assigns to the universal quantifier ‘V’ a set of integers (individuals). 
The same first-order logic may have a relation “even” that assigns “true” for any even numbers. For a certain set of 
algebraic symbols, we may consider a structure of an algebra, e.g., group. 


Definition 4 (Satisfaction) A satisfaction is a relation between structures and formulas. For a structure A and a 
formula ¢, this relation is usually written as AF g. 


For example, a structure of group satisfies a formula “x + (v+z) = (« +y)+z” (associativity). 
We are now in a position to define logic in terms of the above-mentioned components. 


Definition 5 (Logic) A logic is a combined specifications of language, interpretation, and satisfaction. 


The language of a logic is also called syntax. By fixing the syntax, we can tell whether a certain expression is a formula 
of that logic. The remaining elements define the semantics of the logic. By fixing the semantics, we can tell whether 
a certain formula is satisfied by a certain structure. 


‘Tn first-order logic, the distinction between “formulas” and “sentences” is that in the latter, every variable is “bound” by a quantifier. In 
equational logic, all variables in a “formula” are assumed to be universally quantified. 

2For the interpretation of a formula with unbound variable, the “assignment” of variable also needs to be specified. Since we focus on quantified 
formulas, we skip the definition of assignment. 

3Both the language and the structure of a logic may be specified in terms of a signature X, the specification of the involved “sorts” and 
“operations.” In this case, the language and the structure can be represented as L (XZ) and A/g(), called Z-algebra. While the traditional first- 
order logic deals with a single “sort,” i.e., sort of individuals, modern programming specification often uses multiple “sorts,” corresponding to 
multiple (often complex) data types. 


2 Definition of “Theory” 


We now proceed to the definition of “theory” in the context of formal logic. We need to introduce a few more 
definitions about the relation between structures and formulas. 


Definition 6 (Model) For a logic L, a structure A is called a model of formulas ®, if AF 9 for all g € ®. 


By abusing the symbol ‘F’, we also write A F ®. For example, a structure (algebra) of group is a model of formulas 
{x + (vy +z) = («+ y)+2}. The set of all models of ® is written as Mod (®). 


Definition 7 (Logical consequence) For a logic L, a formula g is called a logical consequence of ®, if for each 
Aé Mod (®), AF g. 


By still abusing the symbol ‘F’, we also write ® F g. For example, “x + (vy +z) = (x + y)+2z” is a logical consequence 
of the axioms of group. 

Let us now proceed to define the bare-bone definition of “theory.” Informally, a theory is a set of formulas that is 
closed under logical consequence, or complete with respect to validity. 


Definition 8 (Theory) For a logic L, a theory is a set of formulas ® such that for each formula g, ® F g implies 
ge, 


In other words, if we can derive from a set of formulas a formula that does not belong to the set, the set is not a 
theory. At this point, the only requirement is that the set of formulas is validly closed. It does not even need to be 
consistent. Thus, an inconsistent set of formulas is still a theory as long as it satisfies the above definition. This 
definition of “theory” is good only if there is a mechanism to represent such a closed set. In some cases, we may list 
all the formulas extensionally. But it is not usually the way an interesting theory is described. 

A more concise representation of a theory is based on the idea of axiomatization. Let us proceed with a few more 
definitions necessary for this move. 


Definition 9 (Theory of structures) For a logic L, the theory of aclass of structures C is the set: Th (C) = {og |C F g}. 


Definition 10 (Consequences of formulas) Cn (®) = Th (Mod (®)). 


Definition 11 (Axiomatizable theory) A theory ® is axiomatizable iff there is a decidable set ‘¥ such that ® = 
Cn(¥). 


A theory may or may not be axiomatizable. In addition, the set of axioms may be finite or infinite. Naturally, many 
interesting cases of axiomatization is finite. For example, the theory of group is finitely axiomatizable (in a certain 
logic). This point is particularly important because finite axiomatization allows us to predict beyond what has already 
been described. 


3 Axiomatization via Calculus 


Although the above presentation provides a view about theory sufficient as a background, it is often inconvenient or 
impossible to deal with models in a direct way. Fortunately, the study of logic can provide useful results on the relation 
between semantic analysis via models and syntactic (mechanical) analysis via formulas. 

Let us first introduce the notion of “calculus.” Informally, a calculus is a purely syntactic manipulation of symbols 
to derive a new formula from a set of formulas. 


Definition 12 (Axioms) We consider a finite set of axiom schemata, each of which consists of a decidable subset of 
the formulas of L. 


Here we talk about axiom schemata rather than axiom instances. This is convenient because we can often consolidate 
infinitely many axioms as a single axiom scheme. For simplicity, we may call axiom schemata as axiom. We have 
seen an axiom of group theory. 


Definition 13 (Rules of inference) A rule of inference is a finite, decidable relation between a set of formulas and a 
formula. 


A typical rule of inference is ‘modus ponens’, i.e., from “if p, then g” and “p”, to derive “q”. 
Definition 14 (Calculus) A calculus is a combined specification of axioms and rules of inference. 


Definition 15 (Derivation) Derivation of a formula g from a set of formulas ©® in a calculus is specified by axioms 
and rules of inference, and is written as Dlg. 


Definition 16 (Soundness) A calculus for a logic is sound if ® Fg implies DF g. 


Definition 17 (Completeness) A calculus for a logic is complete if ® F yg implies Og. 


A sound and complete calculus of a logic guarantees that mechanical operations in the calculus is semantically 
valid. In practice, it is often considered sufficient to use a sound calculus (i.e., mechanical operations are always 
correct, but there may not be a proof for some true formula). 

Now, let us return to the topic of axiomatization. An axiomatizable theory can be completely specified by a sound 
and complete calculus. A sound calculus can describe an axiomatizable theory, but the specification may not be 
complete. Thus, the ability of a mechanical specification (axiomatization) of a theory via calculus depends on the 
properties associated with the calculus. 
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